Image classification method and device based on self-supervised attention, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the medical field and the financial field, and discloses a self-supervised attention-based image classification method, device and equipment and a medium, and the method comprises the steps: obtaining an input image, and carrying out the feature extraction of the input image through a convolution feature extractor, and obtaining a feature map; performing self-supervision pre-training on the feature map through a self-supervision attention mechanism to obtain an attention weight matrix; based on the attention weight matrix, utilizing an interpretable classification head to carry out classification prediction on the feature map so as to construct an image classification model; and performing image classification processing on a specified target image by using the image classification model. According to the method, the attention weight is learned in a self-supervision mode, dependence on a large amount of annotation data is reduced, meanwhile, the attention mechanism in the pre-training stage can be directly migrated to the classification task, end-to-end interpretability is achieved, and the accuracy and the interpretability in the image classification task are both improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for image classification based on self-supervised attention. Background Art

[0002] In recent years, deep learning has made significant progress in image classification tasks, but its performance improvements often rely on large amounts of labeled data, which is expensive and time-consuming. To address this problem, self-supervised learning (SSL) has become a research hotspot by designing proxy tasks to learn universal feature representations from unlabeled data. For example, a contrastive predictive coding framework has been proposed to improve feature consistency through momentum contrastive learning, thereby simplifying the contrastive learning process. However, these methods mainly focus on global feature representations and lack explicit modeling of local region relationships, which limits the model's performance in fine-grained classification tasks. For example, for images in the medical field, existing technologies often have difficulty accurately capturing the subtle features of the lesion area, thereby affecting the accuracy of diagnosis. For another example, for images in the financial field, existing technologies have difficulty accurately identifying subtle differences in counterfeit bills, increasing the risk of fraud.

[0003] Meanwhile, the success of the Transformer architecture in computer vision (such as the proposed VisionTransformer) demonstrates the advantages of self-attention mechanisms in capturing long-range dependencies. These methods process images in blocks and calculate inter-block attention weight matrices, dynamically focusing on key regions. However, existing work often relies on supervised training, and the interpretability of their attention mechanisms is limited by the quality and coverage of labeled data. Furthermore, while equal-gradient weighted class activation mapping methods can generate visual explanations, they are typically used as post-processing tools and are not deeply coupled with the model training process, resulting in deviations between the explanations and the model's decision logic. Summary of the Invention

[0004] Embodiments of the present invention provide a method, apparatus, computer device, and storage medium for image classification based on self-supervised attention, aiming to improve the accuracy and interpretability of image classification.

[0005] In a first aspect, an embodiment of the present invention provides an image classification method based on self-supervised attention, comprising:

[0006] Acquire an input image, and perform feature extraction on the input image using a convolutional feature extractor to obtain a feature map;

[0007] Performing self-supervised pre-training on the feature map through a self-supervised attention mechanism to obtain an attention weight matrix;

[0008] Based on the attention weight matrix, using an interpretable classification head to perform classification prediction on the feature map, thereby building an image classification model;

[0009] The image classification model is used to perform image classification processing on the specified target image.

[0010] In a second aspect, an embodiment of the present invention provides an image classification device based on self-supervised attention, comprising:

[0011] A feature extraction unit, configured to obtain an input image and perform feature extraction on the input image using a convolutional feature extractor to obtain a feature map;

[0012] A self-supervised learning unit, configured to perform self-supervised pre-training on the feature map through a self-supervised attention mechanism to obtain an attention weight matrix;

[0013] A classification prediction unit, configured to perform classification prediction on the feature map based on the attention weight matrix using an interpretable classification head, thereby constructing an image classification model;

[0014] The model classification unit is used to perform image classification processing on the specified target image using the image classification model.

[0015] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the image classification method based on self-supervised attention as described in the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image classification method based on self-supervised attention as described in the first aspect.

[0017] An embodiment of the present invention provides an image classification method, apparatus, computer equipment, and storage medium based on self-supervised attention. The method includes: obtaining an input image, and performing feature extraction on the input image using a convolutional feature extractor to obtain a feature map; performing self-supervised pre-training on the feature map using a self-supervised attention mechanism to obtain an attention weight matrix; based on the attention weight matrix, performing classification prediction on the feature map using an interpretable classification head to construct an image classification model; and performing image classification processing on a specified target image using the image classification model. The embodiment of the present invention extracts a feature map from an input image using a convolutional feature extractor, and performs self-supervised learning on the feature map using a self-supervised attention mechanism to obtain an attention weight matrix. Then, based on the attention weight matrix, an interpretable classification head is used to perform classification prediction on the feature map. Through the above process, an image classification model can be constructed, and the target image can be classified using the image classification model. The embodiment of the present invention learns attention weights in a self-supervised manner, reducing dependence on large amounts of labeled data. At the same time, the attention mechanism learned in the pre-training stage can be directly transferred to classification tasks, achieving end-to-end interpretability. The embodiment of the present invention combines self-supervised pre-training with the Transformer attention mechanism to achieve a dual improvement in accuracy and interpretability in image classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of an image classification method based on self-supervised attention provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of a sub-process of an image classification method based on self-supervised attention provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of another sub-process of an image classification method based on self-supervised attention provided by an embodiment of the present invention;

[0022] Figure 4 A schematic block diagram of an image classification device based on self-supervised attention provided by an embodiment of the present invention;

[0023] Figure 5 A schematic block diagram of an image classification device based on self-supervised attention provided by an embodiment of the present invention;

[0024] Figure 6 Another schematic block diagram of an image classification device based on self-supervised attention provided by an embodiment of the present invention;

[0025] Figure 7 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0028] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0029] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0030] The image classification method based on self-supervised attention provided by an embodiment of the present invention can be applied in an application environment where a client and a server interact, wherein the client communicates with the server through a network. The server obtains an input image, and uses a convolutional feature extractor to extract features from the input image to obtain a feature map; the feature map is self-supervised pre-trained by a self-supervised attention mechanism to obtain an attention weight matrix; based on the attention weight matrix, the feature map is classified and predicted using an interpretable classification head to construct an image classification model. The client sends a target image, and the server uses the image classification model to perform image classification processing on the specified target image. The client can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] See below Figure 1 , an embodiment of the present invention provides an image classification method based on self-supervised attention, specifically comprising: steps S101 to S104.

[0032] Step S101: Acquire an input image, and use a convolutional feature extractor to extract features from the input image to obtain a feature map;

[0033] Step S102: performing self-supervised pre-training on the feature map through a self-supervised attention mechanism to obtain an attention weight matrix;

[0034] Step S103: Based on the attention weight matrix, an interpretable classification head is used to perform classification prediction on the feature map to construct an image classification model;

[0035] Step S104: perform image classification processing on the designated target image using the image classification model.

[0036] In this example, a convolutional feature extractor first extracts a feature map from the input image. A self-supervised attention mechanism then performs self-supervised learning on the feature map to generate an attention weight matrix. Next, an interpretable classification head is used to perform classification predictions on the feature map based on the attention weight matrix. This process constructs an image classification model, which can then be used to classify the target image.

[0037] This embodiment learns attention weights through self-supervision, reducing reliance on large amounts of labeled data. Furthermore, the attention mechanism learned during the pre-training phase can be directly transferred to classification tasks, achieving end-to-end interpretability. By combining self-supervised pre-training with the Transformer attention mechanism, this embodiment achieves both improved accuracy and interpretability in image classification tasks. In practical applications, the pre-training phase can use only unlabeled images to learn the attention mechanism, while the fine-tuning phase can utilize a small amount of labeled data to adapt the classification task. Both phases share the same attention calculation logic, ensuring consistency between interpretation results and classification decisions.

[0038] In particular, the self-supervised attention-based image classification method provided in this embodiment is applicable to the medical field. Specifically, this embodiment can accurately classify lesion areas in medical images, thereby improving diagnostic efficiency and accuracy. In medical image analysis, subtle lesion features often determine the diagnostic outcome, but traditional image classification methods often have difficulty capturing this critical information. By introducing a self-supervised attention mechanism, this embodiment can focus on key lesion areas in medical images, accurately capturing and classifying lesion features, and providing doctors with a more reliable auxiliary diagnostic tool. The self-supervised attention-based image classification method provided in this embodiment is also applicable to the financial field. Specifically, this embodiment can accurately identify the authenticity of financial instruments, thereby effectively preventing fraud risks. In the audit of financial instruments, the subtle differences in counterfeit instruments are often difficult to detect with the naked eye, and traditional image classification methods perform poorly for such fine-grained classification tasks. By combining self-supervised learning and the Transformer attention mechanism, this embodiment can accurately identify signs of counterfeiting in instruments, improving the security of financial transactions. Of course, the image classification method provided in this embodiment can also be applied to other fields, such as security monitoring and autonomous driving. For example, in the security monitoring field, accurate classification of targets in surveillance videos can enable intelligent early warning and behavior recognition, improving safety and prevention capabilities. In the field of autonomous driving, accurate classification of vehicles, pedestrians, traffic signs, and other road scenes can improve the safety and reliability of autonomous driving systems.

[0039] In one embodiment, the step of acquiring an input image and performing feature extraction on the input image using a convolutional feature extractor to obtain a feature map includes:

[0040] According to the following formula, the feature map is obtained by extracting features from the input image using a convolutional neural network:

[0041]

[0042] Wherein, F represents the feature map, CNN represents the convolutional neural network, X represents the input image, h and w represent the spatial height and width of the feature map respectively, and d represents the feature dimension.

[0043] The convolutional feature extractor described in this embodiment uses a standard CNN architecture (such as ResNet50) as the backbone network, which is responsible for converting the input image into a high-dimensional feature representation. The feature map output by the convolutional feature extraction will maintain the spatial structure of the original image, thus providing a basis for subsequent attention calculations. Specifically, given an input image Where H and W represent the image height and width respectively, and C is the number of channels (for RGB images, C=3). The feature extractor outputs a feature map:

[0044]

[0045] Here, h×w is the spatial size of the feature map (usually h = H / 32, w = W / 32), and d is the feature dimension (e.g., d = 2048 for ResNet50). The feature map preserves the spatial hierarchical structure of the original image, with each spatial location corresponding to a local receptive field of the input image.

[0046] Of course, in specific embodiments, different convolutional neural network architectures can be used as feature extractors to adapt to different application scenarios and data characteristics. For example, in medical image analysis, a deeper network architecture may be required to capture more subtle lesion features; in financial image detection, a lightweight network architecture may be required to improve processing speed and efficiency. The choice of feature extractor should be based on a trade-off between the requirements of the specific task and the characteristics of the data to ensure optimal performance; in the field of security monitoring, considering the real-time requirements, a lighter network architecture can be selected. In addition, the design of the feature extractor can also be combined with specific task requirements to introduce specific prior knowledge or regularization strategies to improve the effectiveness and robustness of the feature representation.

[0047] In one embodiment, if Figure 2 As shown, the step S102 includes steps S201 to S203.

[0048] Step S201: Divide the feature map into N non-overlapping patches Among them, P i represents the i-th non-overlapping patch;

[0049] Step S202: According to the following formula, the feature map is patched to zero using a randomly selected mask ratio to obtain a mask feature map;

[0050]

[0051] in, Represents the mask feature map, Mask represents the mask, F represents the feature map, M represents the mask position index set, and ρ represents the mask ratio;

[0052] Step S203: Use a Transformer encoder to encode the mask feature map to obtain an attention weight matrix.

[0053] The self-supervised pre-training described in this embodiment includes three parts: patch division, random masking, and Transformer encoder. This embodiment designs a mask patch reconstruction task to learn an attention weight matrix with spatial perception capabilities under unsupervised conditions, and these weights can be directly used to explain the decision basis of the model in subsequent classification tasks. Specifically, when performing self-supervised pre-training, this embodiment first performs patch division on the feature map, that is, divides the feature map F into N = h × w non-overlapping patches Each patch Corresponding to a spatial position in the feature map. These patches are then zeroed according to a randomly selected mask ratio (e.g., setting ρ = 0.6), resulting in a corresponding masked feature map. Finally, the masked feature map is encoded using a Transformer encoder to obtain an attention weight matrix.

[0054] Here, the Transformer encoder is used to encode the mask feature map to obtain the attention weight matrix, which specifically includes:

[0055] Perform linear projection on the mask feature map to obtain the query matrix Q, key matrix K and value matrix V:

[0056]

[0057] in, All are learnable parameters;

[0058] The weight matrix is ​​calculated using multi-head attention to obtain the attention weight matrix A:

[0059]

[0060] Among them, d k Represents the projection dimension.

[0061] The Transformer encoder described in this embodiment is a deep learning model based on the self-attention mechanism. It captures the relationship between the elements in the sequence by encoding the input sequence. In this embodiment, the standard Transformer structure is used to process the mask feature map. First, the mask feature map is linearly projected to obtain the query matrix Q, the key matrix K, and the value matrix V. Then, the multi-head attention mechanism is used to calculate the weight matrix to obtain the attention weight matrix A. Among them, the linear projection involves learnable parameters, and the multi-head attention mechanism performs a softmax calculation based on the transpose of the query matrix Q and the key matrix K to obtain the N×N attention weight matrix A. By using the Transformer encoder in this embodiment, the model can capture long-range dependencies in the feature map, improving the accuracy and robustness of image classification. At the same time, because the Transformer encoder has the ability to parallelize computing, it can greatly improve the training speed and inference speed of the model, making the image classification method provided in this embodiment more efficient in practical applications.

[0062] Furthermore, in one embodiment, after the step of encoding the mask feature map using a Transformer encoder to obtain an attention weight matrix, the following steps are included:

[0063] The decoder is used to predict the original features of the masked patch of the feature map to obtain the predicted features:

[0064]

[0065] in, represents the prediction feature, MLP represents the support vector machine, z i ∈Z, Z represents the weighted sum of the attention weight matrix and the value matrix,

[0066] According to the following formula, the reconstruction loss is constructed using the mean square error:

[0067]

[0068] Among them, L rec represents the reconstruction loss, P i Represents the true features of the feature map;

[0069] The Transformer encoder is parameter updated using the reconstruction loss.

[0070] This embodiment predicts the original features of the masked patch through the decoder. First, the weighted result Z is obtained by weighting the attention weight matrix A and the value matrix:

[0071]

[0072] Then design a decoder (for example, use a 2-layer MLP as a decoder) to predict the original features of the masked patch:

[0073]

[0074] The mean squared error is then used to construct the reconstruction loss to measure the difference between the predicted features and the true features. Finally, the reconstruction loss is used to update the parameters of the Transformer encoder.

[0075] It should be noted that this embodiment effectively improves the model's ability to recognize and perceive local features by performing patch-level reconstruction. This approach is particularly suitable for situations requiring detailed and sophisticated analysis, such as in the field of medical image recognition, where it can help the system more accurately identify key details and subtle differences in images.

[0076] In one embodiment, if Figure 3 As shown, the step S103 includes: steps S301 to S303.

[0077] Step S301: performing global average pooling feature calculation on the feature map to obtain corresponding global average pooling features;

[0078]

[0079] Among them, f gap represents the global average pooling feature, P i Non-overlapping patches, N represents the number of patches;

[0080] Step S302: Based on the attention weight matrix, calculate the attention weighted features of the feature map to obtain corresponding attention weighted features;

[0081]

[0082] Among them, f att represents the attention weighted feature, α i Represents element A ij The cumulative sum of A ij Represents the element in row i and column j in the attention weight matrix A;

[0083] Step S303: Concatenate the global average pooling features and the attention weighted features and input them into a classifier, which then outputs the classification result:

[0084] y=softmax(W c [f gap ;f att ])

[0085] Among them, y represents the classification result, W c Represents the classification weight.

[0086] The interpretable classification head described in this embodiment consists of a feature weighting layer and a fully connected classifier. During the fine-tuning phase, the pre-trained attention weight matrix is ​​directly reused to spatially weight the feature map. This involves first calculating the global average pooling feature for the complete feature map, and then simultaneously calculating the attention-weighted feature. Finally, these two features are concatenated and input into the classifier, which outputs the classification result through the fully connected layer. During the attention-weighted feature calculation process, the importance of each patch is determined by its corresponding attention weight, enabling the model to focus on patch regions that have a significant impact on the classification result. In this way, this embodiment not only improves classification accuracy but also enhances model interpretability. Specifically, classification decisions can be visualized based on the attention weight matrix, helping users understand why the model makes a specific classification. Furthermore, this embodiment combines global average pooling features with attention-weighted features to further enrich feature representation and improve the model's generalization capability. In terms of classifier design, this embodiment uses a softmax function as the output layer to output multi-category probability distributions, thereby enabling multi-classification tasks. During the fine-tuning phase, the model can be trained using a small amount of labeled data to adapt to specific application scenarios. In this way, the self-supervised attention-based image classification method provided in this embodiment can achieve end-to-end interpretability while maintaining high accuracy.

[0087] Furthermore, the image classification method based on self-supervised attention further includes:

[0088] Based on the size of the input image, the cumulative sum α i Perform upsampling to obtain the attention heat map.

[0089] The attention heat map can intuitively show the contribution of each area in the input image to the classification decision. Specifically, the highlighted areas in the attention heat map represent image areas that have a significant impact on the classification results. These areas usually correspond to key lesion features, target objects, or important scenes in the image. For example, in medical scenarios, by visualizing the attention heat map, doctors or analysts can more easily understand why the model makes a specific classification, thereby improving the accuracy and efficiency of diagnosis. In addition, the attention heat map can also serve as an additional diagnostic basis to assist doctors in locating and identifying lesion areas. Therefore, in fields such as medical image analysis, security monitoring, and autonomous driving, this visualization tool is of great significance. It can not only improve the interpretability of the model, but also enhance users' trust and satisfaction with the model's decisions.

[0090] Figure 4A schematic block diagram of an image classification device 400 based on self-supervised attention provided in an embodiment of the present invention, wherein the device 400 includes:

[0091] The feature extraction unit 401 is used to obtain an input image and perform feature extraction on the input image using a convolutional feature extractor to obtain a feature map;

[0092] A self-supervised learning unit 402 is configured to perform self-supervised pre-training on the feature map through a self-supervised attention mechanism to obtain an attention weight matrix;

[0093] A classification prediction unit 403 is configured to perform classification prediction on the feature map based on the attention weight matrix using an interpretable classification head, thereby constructing an image classification model;

[0094] The model classification unit 404 is configured to perform image classification processing on the designated target image using the image classification model.

[0095] In this example, a convolutional feature extractor first extracts a feature map from the input image. A self-supervised attention mechanism then performs self-supervised learning on the feature map to generate an attention weight matrix. Next, an interpretable classification head is used to perform classification predictions on the feature map based on the attention weight matrix. This process constructs an image classification model, which can then be used to classify the target image.

[0096] This embodiment learns attention weights through self-supervision, reducing reliance on large amounts of labeled data. Furthermore, the attention mechanism learned during the pre-training phase can be directly transferred to classification tasks, achieving end-to-end interpretability. By combining self-supervised pre-training with the Transformer attention mechanism, this embodiment achieves both improved accuracy and interpretability in image classification tasks. In practical applications, the pre-training phase can use only unlabeled images to learn the attention mechanism, while the fine-tuning phase can utilize a small amount of labeled data to adapt the classification task. Both phases share the same attention calculation logic, ensuring consistency between interpretation results and classification decisions.

[0097] In particular, the self-supervised attention-based image classification method provided in this embodiment is applicable to the medical field. Specifically, this embodiment can accurately classify lesion areas in medical images, thereby improving diagnostic efficiency and accuracy. In medical image analysis, subtle lesion features often determine the diagnostic outcome, but traditional image classification methods often have difficulty capturing this critical information. By introducing a self-supervised attention mechanism, this embodiment can focus on key lesion areas in medical images, accurately capturing and classifying lesion features, and providing doctors with a more reliable auxiliary diagnostic tool. The self-supervised attention-based image classification method provided in this embodiment is also applicable to the financial field. Specifically, this embodiment can accurately identify the authenticity of financial instruments, thereby effectively preventing fraud risks. In the audit of financial instruments, the subtle differences in counterfeit instruments are often difficult to detect with the naked eye, and traditional image classification methods perform poorly for such fine-grained classification tasks. By combining self-supervised learning and the Transformer attention mechanism, this embodiment can accurately identify signs of counterfeiting in instruments, improving the security of financial transactions. Of course, the image classification method provided in this embodiment can also be applied to other fields, such as security monitoring and autonomous driving. For example, in the security monitoring field, accurate classification of targets in surveillance videos can enable intelligent early warning and behavior recognition, improving safety and prevention capabilities. In the field of autonomous driving, accurate classification of vehicles, pedestrians, traffic signs, and other road scenes can improve the safety and reliability of autonomous driving systems.

[0098] In one embodiment, the feature extraction unit 401 includes:

[0099] The convolution extraction unit is configured to extract features from the input image using a convolutional neural network according to the following formula to obtain the feature map:

[0100]

[0101] Wherein, F represents the feature map, CNN represents the convolutional neural network, X represents the input image, h and w represent the spatial height and width of the feature map respectively, and d represents the feature dimension.

[0102] The convolutional feature extractor described in this embodiment uses a standard CNN architecture (such as ResNet50) as the backbone network, which is responsible for converting the input image into a high-dimensional feature representation. The feature map output by the convolutional feature extraction will maintain the spatial structure of the original image, thus providing a basis for subsequent attention calculations. Specifically, given an input image Where H and W represent the image height and width respectively, and C is the number of channels (for RGB images, C=3). The feature extractor outputs a feature map:

[0103]

[0104] Here, h×w is the spatial size of the feature map (usually h = H / 32, w = W / 32), and d is the feature dimension (e.g., d = 2048 for ResNet50). The feature map preserves the spatial hierarchical structure of the original image, with each spatial location corresponding to a local receptive field of the input image.

[0105] Of course, in specific embodiments, different convolutional neural network architectures can be used as feature extractors to adapt to different application scenarios and data characteristics. For example, in medical image analysis, a deeper network architecture may be required to capture more subtle lesion features; in financial image detection, a lightweight network architecture may be required to improve processing speed and efficiency. The choice of feature extractor should be based on a trade-off between the requirements of the specific task and the characteristics of the data to ensure optimal performance; in the field of security monitoring, considering the real-time requirements, a lighter network architecture can be selected. In addition, the design of the feature extractor can also be combined with specific task requirements to introduce specific prior knowledge or regularization strategies to improve the effectiveness and robustness of the feature representation.

[0106] In one embodiment, if Figure 5 As shown, the self-supervised learning unit 402 includes:

[0107] The patch division unit 501 is used to divide the feature map into N non-overlapping patches Among them, P i represents the i-th non-overlapping patch;

[0108] A mask setting unit 502 is configured to perform patching and zeroing on the feature map using a randomly selected mask ratio according to the following formula to obtain a mask feature map;

[0109]

[0110] in, Represents the mask feature map, Mask represents the mask, F represents the feature map, M represents the mask position index set, and ρ represents the mask ratio;

[0111] The feature encoding unit 503 is used to encode the mask feature map using a Transformer encoder to obtain an attention weight matrix.

[0112] The self-supervised pre-training described in this embodiment includes three parts: patch division, random masking, and Transformer encoder. This embodiment designs a mask patch reconstruction task to learn an attention weight matrix with spatial perception capabilities under unsupervised conditions, and these weights can be directly used to explain the decision basis of the model in subsequent classification tasks. Specifically, when performing self-supervised pre-training, this embodiment first performs patch division on the feature map, that is, divides the feature map F into N = h × w non-overlapping patches Each patch Corresponding to a spatial position in the feature map. These patches are then zeroed according to a randomly selected mask ratio (e.g., setting ρ = 0.6), resulting in a corresponding masked feature map. Finally, the masked feature map is encoded using a Transformer encoder to obtain an attention weight matrix.

[0113] In one embodiment, the feature encoding unit 503 includes:

[0114] A linear projection unit is used to perform linear projection on the mask feature map to obtain a query matrix Q, a key matrix K, and a value matrix V:

[0115]

[0116] in, All are learnable parameters;

[0117] The weight calculation unit is used to calculate the weight matrix using multi-head attention to obtain the attention weight matrix A:

[0118]

[0119] Among them, d k Represents the projection dimension.

[0120] The Transformer encoder described in this embodiment is a deep learning model based on the self-attention mechanism. It captures the relationship between the elements in the sequence by encoding the input sequence. In this embodiment, the standard Transformer structure is used to process the mask feature map. First, the mask feature map is linearly projected to obtain the query matrix Q, the key matrix K, and the value matrix V. Then, the multi-head attention mechanism is used to calculate the weight matrix to obtain the attention weight matrix A. Among them, the linear projection involves learnable parameters, and the multi-head attention mechanism performs a softmax calculation based on the transpose of the query matrix Q and the key matrix K to obtain the N×N attention weight matrix A. By using the Transformer encoder in this embodiment, the model can capture long-range dependencies in the feature map, improving the accuracy and robustness of image classification. At the same time, because the Transformer encoder has the ability to parallelize computing, it can greatly improve the training speed and inference speed of the model, making the image classification method provided in this embodiment more efficient in practical applications.

[0121] In one embodiment, the self-supervised attention-based image classification apparatus 400 further includes:

[0122] The decoding prediction unit is used to use the decoder to predict the original features of the masked patch of the feature map to obtain the predicted features:

[0123]

[0124] in, represents the prediction feature, MLP represents the support vector machine, z i ∈Z, Z represents the weighted sum of the attention weight matrix and the value matrix,

[0125] The loss construction unit is used to construct the reconstruction loss using the mean square error according to the following formula:

[0126]

[0127] Among them, L rec represents the reconstruction loss, P i Represents the true features of the feature map;

[0128] A parameter updating unit is used to update the parameters of the Transformer encoder using the reconstruction loss.

[0129] This embodiment predicts the original features of the masked patch through the decoder. First, the weighted result Z is obtained by weighting the attention weight matrix A and the value matrix:

[0130]

[0131] Then design a decoder (for example, use a 2-layer MLP as a decoder) to predict the original features of the masked patch:

[0132]

[0133] The mean squared error is then used to construct the reconstruction loss to measure the difference between the predicted features and the true features. Finally, the reconstruction loss is used to update the parameters of the Transformer encoder.

[0134] It should be noted that this embodiment effectively improves the model's ability to recognize and perceive local features by performing patch-level reconstruction. This approach is particularly suitable for situations requiring detailed and sophisticated analysis, such as in the field of medical image recognition, where it can help the system more accurately identify key details and subtle differences in images.

[0135] In one embodiment, if Figure 6 As shown, the classification prediction unit 403 includes:

[0136] A global pooling unit 601 is used to perform global average pooling feature calculation on the feature map to obtain corresponding global average pooling features;

[0137]

[0138] Among them, f gap represents the global average pooling feature, P i Non-overlapping patches, N represents the number of patches;

[0139] A weighted calculation unit 602 is configured to perform attention weighted feature calculation on the feature map based on the attention weight matrix to obtain corresponding attention weighted features;

[0140]

[0141] Among them, f att represents the attention weighted feature, α i Represents element A ij The cumulative sum of A ij Represents the element in row i and column j in the attention weight matrix A;

[0142] The splicing classification unit 603 is used to splice the global average pooling feature and the attention weighted feature, and input them into the classifier, which then outputs the classification result:

[0143] y=softmax(W c [f gap ;f att ])

[0144] Among them, y represents the classification result, W c Represents the classification weight.

[0145] The interpretable classification head described in this embodiment consists of a feature weighting layer and a fully connected classifier. During the fine-tuning phase, the pre-trained attention weight matrix is ​​directly reused to spatially weight the feature map. This involves first calculating the global average pooling feature for the complete feature map, and then simultaneously calculating the attention-weighted feature. Finally, these two features are concatenated and input into the classifier, which outputs the classification result through the fully connected layer. During the attention-weighted feature calculation process, the importance of each patch is determined by its corresponding attention weight, enabling the model to focus on patch regions that have a significant impact on the classification result. In this way, this embodiment not only improves classification accuracy but also enhances model interpretability. Specifically, classification decisions can be visualized based on the attention weight matrix, helping users understand why the model makes a specific classification. Furthermore, this embodiment combines global average pooling features with attention-weighted features to further enrich feature representation and improve the model's generalization capability. In terms of classifier design, this embodiment uses a softmax function as the output layer to output multi-category probability distributions, thereby enabling multi-classification tasks. During the fine-tuning phase, the model can be trained using a small amount of labeled data to adapt to specific application scenarios. In this way, the self-supervised attention-based image classification method provided in this embodiment can achieve end-to-end interpretability while maintaining high accuracy.

[0146] In one embodiment, the self-supervised attention-based image classification apparatus 400 further includes:

[0147] A sampling processing unit is used to process the accumulated sum α based on the size of the input image. i Perform upsampling to obtain the attention heat map.

[0148] The attention heat map can intuitively show the contribution of each area in the input image to the classification decision. Specifically, the highlighted areas in the attention heat map represent image areas that have a significant impact on the classification results. These areas usually correspond to key lesion features, target objects, or important scenes in the image. For example, in medical scenarios, by visualizing the attention heat map, doctors or analysts can more easily understand why the model makes a specific classification, thereby improving the accuracy and efficiency of diagnosis. In addition, the attention heat map can also serve as an additional diagnostic basis to assist doctors in locating and identifying lesion areas. Therefore, in fields such as medical image analysis, security monitoring, and autonomous driving, this visualization tool is of great significance. It can not only improve the interpretability of the model, but also enhance users' trust and satisfaction with the model's decisions.

[0149] See also Figure 7, Figure 7 This is a schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device is a device with wireless communication and wired communication.

[0150] The computer device includes a processor 702 , a memory, and a network interface 705 connected via a system bus 701 , wherein the memory may include a non-volatile storage medium 703 and an internal memory 704 .

[0151] The non-volatile storage medium 703 may store an operating system 7031 and a computer program 7032. When the computer program 7032 is executed, the processor 702 may perform an image classification method based on self-supervisory attention.

[0152] The processor 702 is used to provide computing and control capabilities to support the operation of the entire computer device.

[0153] The internal memory 704 provides an environment for the operation of the computer program 7032 in the non-volatile storage medium 703. When the computer program 7032 is executed by the processor 702, the processor 702 can perform an image classification method based on self-supervised attention.

[0154] The network interface 705 is used to communicate with other devices over the network. Figure 7 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0155] The processor 702 is configured to run a computer program 7032 stored in the memory to implement any embodiment of the above-mentioned self-supervised attention-based image classification method.

[0156] It should be understood that in the embodiment of the present invention, the processor 702 may be a central processing unit (CPU), and the processor 702 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0157] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0158] Acquire an input image, and perform feature extraction on the input image using a convolutional feature extractor to obtain a feature map;

[0159] Performing self-supervised pre-training on the feature map through a self-supervised attention mechanism to obtain an attention weight matrix;

[0160] Based on the attention weight matrix, using an interpretable classification head to perform classification prediction on the feature map, thereby building an image classification model;

[0161] The image classification model is used to perform image classification processing on the specified target image.

[0162] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When executed, the computer program can implement the steps provided in the above embodiments. The storage medium can include a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0163] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0164] Acquire an input image, and perform feature extraction on the input image using a convolutional feature extractor to obtain a feature map;

[0165] Performing self-supervised pre-training on the feature map through a self-supervised attention mechanism to obtain an attention weight matrix;

[0166] Based on the attention weight matrix, using an interpretable classification head to perform classification prediction on the feature map, thereby building an image classification model;

[0167] The image classification model is used to perform image classification processing on the specified target image.

[0168] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0169] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0170] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0171] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A self-supervised attention-based image classification method, characterized in that include: Acquire an input image, and perform feature extraction on the input image using a convolutional feature extractor to obtain a feature map; Performing self-supervised pre-training on the feature map through a self-supervised attention mechanism to obtain an attention weight matrix; Based on the attention weight matrix, using an interpretable classification head to perform classification prediction on the feature map, thereby building an image classification model; The image classification model is used to perform image classification processing on the specified target image.

2. The image classification method based on self-supervised attention according to claim 1, characterized in that The step of acquiring an input image and performing feature extraction on the input image using a convolutional feature extractor to obtain a feature map includes: According to the following formula, the feature map is obtained by extracting features from the input image using a convolutional neural network: Wherein, F represents the feature map, CNN represents the convolutional neural network, X represents the input image, h and w represent the spatial height and width of the feature map respectively, and d represents the feature dimension.

3. The image classification method based on self-supervised attention according to claim 1, characterized in that The self-supervised pre-training of the feature map is performed through the self-supervised attention mechanism to obtain an attention weight matrix, including: Divide the feature map into N non-overlapping patches Among them, P i represents the i-th non-overlapping patch; According to the following formula, the feature map is patched to zero using a randomly selected mask ratio to obtain a mask feature map; in, Represents the mask feature map, Mask represents the mask, F represents the feature map, M represents the mask position index set, and ρ represents the mask ratio; The mask feature map is encoded using a Transformer encoder to obtain an attention weight matrix.

4. The image classification method based on self-supervised attention according to claim 3, characterized in that The Transformer encoder is used to encode the mask feature map to obtain an attention weight matrix, including: Perform linear projection on the mask feature map to obtain the query matrix Q, key matrix K and value matrix V: in, All are learnable parameters; The weight matrix is ​​calculated using multi-head attention to obtain the attention weight matrix A: Among them, d k Represents the projection dimension.

5. The image classification method based on self-supervised attention according to claim 4, characterized in that After the step of encoding the mask feature map using a Transformer encoder to obtain an attention weight matrix, the following steps are included: The decoder is used to predict the original features of the masked patch of the feature map to obtain the predicted features: in, represents the prediction feature, MLP represents the support vector machine, z i ∈Z, Z represents the weighted sum of the attention weight matrix and the value matrix, According to the following formula, the reconstruction loss is constructed using the mean square error: Among them, L rec represents the reconstruction loss, P i Represents the true features of the feature map; The Transformer encoder is parameter updated using the reconstruction loss.

6. The image classification method based on self-supervised attention according to claim 3, characterized in that The method of performing classification prediction on the feature map based on the attention weight matrix using an interpretable classification head includes: Performing global average pooling feature calculation on the feature map to obtain corresponding global average pooling features; Among them, f gap represents the global average pooling feature, P i Non-overlapping patches, N represents the number of patches; Based on the attention weight matrix, performing attention weighted feature calculation on the feature map to obtain corresponding attention weighted features; Among them, f att represents the attention weighted feature, α i Represents element A ij The cumulative sum of A ij Represents the element in row i and column j in the attention weight matrix A; The global average pooling features and attention weighted features are concatenated and input into the classifier, which then outputs the classification result: y=softmax(W c [f gap ;f att ]) Among them, y represents the classification result, W c Represents the classification weight.

7. The image classification method based on self-supervised attention according to claim 6, characterized in that Also includes: Based on the size of the input image, the cumulative sum α i Perform upsampling to obtain the attention heat map.

8. An image classification device based on self-supervised attention, characterized in that include: A feature extraction unit, configured to obtain an input image and perform feature extraction on the input image using a convolutional feature extractor to obtain a feature map; A self-supervised learning unit, configured to perform self-supervised pre-training on the feature map through a self-supervised attention mechanism to obtain an attention weight matrix; A classification prediction unit, configured to perform classification prediction on the feature map based on the attention weight matrix using an interpretable classification head, thereby constructing an image classification model; The model classification unit is used to perform image classification processing on the specified target image using the image classification model.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image classification method based on self-supervised attention is implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image classification method based on self-supervised attention is implemented as described in any one of claims 1 to 7.

Images