Segmentation-guided image classification method and system based on hierarchical attention mechanism

Abstract

The application relates to a segmentation-guided image classification method and system based on a hierarchical attention mechanism, a multi-level feature coordination mechanism is designed, a bidirectional mask calibration method is adopted to suppress irrelevant background noise interference in an image, recover missed foreground features, and then extract ROI feature maps of different granularities from different depths of an encoder; then, a discriminative sampling attention mechanism is introduced, discriminative features are sampled from each granularity ROI feature map in parallel, and an enhanced ROI feature map corresponding to each granularity and a spatial feature weight map of the discriminative features are generated; finally, a multi-granularity weighted fusion strategy is adopted to fuse the enhanced ROI feature maps of different granularities, so that comprehensive judgment can be made according to details and high-level semantics at the same time, the shortcomings of single-granularity feature information are avoided, and more comprehensive and more accurate classification decisions are made.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation, in particular to a segmentation-guided image classification method and system based on a hierarchical attention mechanism. BACKGROUND

[0002] Medical image classification plays an important role in computer-aided diagnosis (CAD) systems, providing key support for disease detection, prognosis evaluation, and clinical decision-making. Through deep learning models, especially convolutional neural networks (CNNs), multi-level features can be automatically learned from complex medical images, improving diagnostic accuracy and reducing manual workload. However, existing image-level classification methods based on whole-image features have the following problems:

[0003] Susceptible to irrelevant background interference: In the case of small lesions or low contrast, diagnostic-related signals are easily overwhelmed by background noise, reducing classification robustness;

[0004] Lack of precise positioning ability: It is difficult to accurately identify the specific area leading to classification results, reducing the model's explainability and physician's trust;

[0005] Insufficient use of regional information: Although some methods introduce segmentation encoding to guide classification, most only perform feature extraction at the ROI (region of interest) level, ignoring more fine-grained discriminative sub-regions within the ROI. Moreover, directly applying image space encoding to feature space introduces representation level differences and mask errors, affecting classification performance. SUMMARY

[0006] The technical problem to be solved by the present application is how to improve the robustness, accuracy, and explainability of medical image classification.

[0007] The present application provides a segmentation-guided image classification method based on a hierarchical attention mechanism, comprising:

[0008] Step 1: Obtain the image to be processed;

[0009] Step 2: Use an encoder-decoder network to extract multi-level feature maps from the input image; decode the deepest feature map to generate an initial segmentation probability map;

[0010] Step 3: Based on the initial segmentation probability map, use a bidirectional mask calibration method to reconstruct error-corrected and semantically aligned ROI attention maps in the feature space;

[0011] Step 4: Based on the ROI attention map, spatially weight the multi-level feature maps to extract multi-granularity ROI feature maps;

[0012] Step 5: Based on the discriminative sampling attention mechanism, generate enhanced ROI feature maps and spatial feature weight maps of discriminative features corresponding to each granularity on the multi-granularity ROI feature maps.

[0013] Step 6: Through a multi-granularity weighted fusion strategy, aggregate the spatial feature weight maps and fuse the enhanced ROI feature maps to obtain a fused feature vector;

[0014] Step 7: Perform image classification based on the fused feature vector.

[0015] Compared with the prior art, this application has the following advantages: This application designs a multi-level feature collaboration mechanism, which uses a bidirectional mask calibration method to suppress irrelevant background noise interference in the image, recover the missed foreground features, and then extracts ROI feature maps of different granularities from the encoder; then, by introducing a discriminative sampling attention mechanism, discriminative features are sampled in parallel from each granularity ROI feature map, and the enhanced ROI feature map and the spatial feature weight map of the discriminative features corresponding to each granularity are generated; finally, a multi-granularity weighted fusion strategy is used to fuse the enhanced ROI feature maps of different granularities, so that comprehensive judgment can be made based on detailed texture and high-level semantics at the same time, avoiding the insufficiency of single-granularity feature information, thereby making a more comprehensive and accurate classification decision.

[0016] In one possible implementation, the initial segmentation probability map includes a foreground probability map and a background probability map, and step 3 specifically includes:

[0017] Step 301: Element-wise multiplication is used to calculate the background feature map from the background probability map and the foreground feature map from the foreground probability map.

[0018] Step 302: Sample high-confidence background feature vectors from the background feature map, and calculate the semantic similarity between each background feature vector and the foreground feature map to obtain the response map of the background false detection region to be suppressed;

[0019] Step 303: Sample high-confidence foreground feature vectors from the foreground feature map, calculate the semantic similarity between each foreground feature vector and the background feature map, apply a Gaussian distance decay term to each semantic similarity, and obtain the response map of the foreground missed detection region to be recovered.

[0020] Step 304: The foreground probability map, the foreground missed detection region response map, and the background false detection region response map are fused to generate the ROI attention map.

[0021] Compared with existing technologies, the above-mentioned technical solution can suppress background noise and artifacts that are mistakenly classified as foreground in coarse segmentation; and effectively find and restore small or blurry foreground regions that are missed in coarse segmentation, ensuring the integrity of key information, purifying ROI features, and improving the signal-to-noise ratio of subsequent processing.

[0022] In one possible implementation, when sampling the high-confidence background feature vector in step 302 and the high-confidence foreground feature vector in step 303, a lightweight learnable multilayer bandpass resampler is used to resample the initial segmentation probability map.

[0023] After sampling high-confidence background feature vectors and high-confidence foreground feature vectors, the sampled background feature vectors and foreground feature vectors are filtered by a binary gating function.

[0024] In one possible implementation, sampling the high-confidence background feature vector specifically includes:

[0025] Select the highest confidence level from the background probability map of the initial segmentation probability map after resampling. 1 pixel;

[0026] Based on selection The position of each pixel is located from the background feature map, and the corresponding background feature vector is sampled.

[0027] Dynamically set to the number of background pixels in the initial segmentation probability map ;

[0028] The sampling of high-confidence foreground feature vectors specifically includes:

[0029] In the foreground probability map of the initial segmentation probability map after resampling, select the foreground with the highest confidence. 1 pixel;

[0030] Based on selection The position of each pixel is located from the foreground feature map, and the corresponding foreground feature vector is sampled.

[0031] Dynamically set to the number of foreground pixels in the initial segmentation probability map .

[0032] In one possible implementation, step 5 specifically includes:

[0033] Step 501: The multi-granularity ROI feature map outputs probabilistic sampling maps of each granularity through linear mapping and the Softmax function;

[0034] Step 502: Based on the probability weights of each channel in the sampling map of each granularity, perform a weighted summation of all spatial locations in the multi-granularity ROI feature map to obtain the discriminative feature descriptor.

[0035] Step 503: Apply a random discard operation to the discriminative feature descriptors to obtain the filtered discriminative feature descriptors;

[0036] Step 504: The highest-granularity ROI feature map is processed by a learnable weight matrix. Mapped to query space The selected discriminative feature descriptors are then processed using a learnable weight matrix. , Mapped to bond space respectively Value space ;

[0037] Step 505, calculate the query space Bond space The similarity is used to obtain the attention weight matrix;

[0038] Step 506, based on the attention weight matrix, adjust the value space. Perform a weighted summation to obtain a high-level ROI feature map enhanced by the current granularity discriminative features;

[0039] Step 507: Perform global average pooling compression on the discriminative feature descriptors to obtain a set of scalars; input all scalars into the FFN network and activate them through the Sigmoid activation function to obtain the importance score of each discriminative feature;

[0040] Step 508: Perform column-by-column Softmax normalization on each column of the attention weight matrix; and combine the importance score to perform a weighted summation on the normalized attention columns to obtain a spatial feature weight map for highlighting the discrimination region.

[0041] Step 509: The high-level ROI feature map and the spatial feature weight map are further fused by weighted addition. After adjusting the dimensions through the projection matrix of multi-head attention, an enhanced ROI feature map is obtained.

[0042] Compared with existing technologies, the above-mentioned technical solution can proactively and adaptively extract the most discriminative sub-region features (discriminative feature descriptors) from features of various granularities, rather than passively accepting all features; and effectively inject discriminative information into high-level features to enhance representation, and generate an interpretable spatial weight map that indicates the location of the decision-making basis.

[0043] In one possible implementation, the calculation expression for step 505 is:

[0044] ;

[0045] In the formula, Represents the attention weight matrix; Indicates position code, Indicates reservation A discriminative feature descriptor, Indicates the first The number of channels in a ROI feature map. Indicates matrix transpose. This represents the height and width of the 4th ROI feature map;

[0046] Step 506 calculates the expression as follows:

[0047] ;

[0048] In the formula, This represents the high-level ROI feature map.

[0049] In one possible implementation, the calculation expression for step 507 is:

[0050] ;

[0051] In the formula, Indicates importance score, This represents the discriminative feature descriptor after filtering. Used to The channel is compressed to 1. The network consists of two fully connected layers and an intermediate normalization layer. Indicates the activation function;

[0052] The calculation expression for step 508 is:

[0053] ;

[0054] In the formula, The column index representing the attention weight matrix;

[0055] The calculation expression for step 509 is:

[0056] ;

[0057] In the formula, This represents an enhanced ROI feature map. Represents the spatial feature weight map. The projection matrix representing multi-head attention. This represents the learnable scaling factor.

[0058] In one possible implementation, step 6 specifically includes:

[0059] Step 601: Concatenate the enhanced ROI feature maps generated at each granularity to form a multi-granularity feature map;

[0060] Step 602: Weight the spatial feature weight maps generated at each granularity to form a comprehensive feature weight map;

[0061] Step 603: Based on the comprehensive feature weight map, perform a global weighted pooling operation on the multi-granularity feature map to obtain the fused feature vector.

[0062] Compared with existing technologies, the above-mentioned technical solution can effectively integrate multi-scale and multi-granular discriminative information from micro to macro, making image classification decisions depend on both local details and global context. By using global weighted pooling (GWP) instead of ordinary pooling, the contribution of discriminative regions to the final decision is strengthened, while the influence of non-discriminative regions is weakened.

[0063] A segmentation-guided image segmentation system, used to perform the segmentation-guided image classification method based on hierarchical attention mechanism described in any one of the preceding claims, comprising:

[0064] The image acquisition module is used to acquire the image to be processed.

[0065] The encoder-decoder network module, connected to the image acquisition module, is used to extract multi-level feature maps of the image and generate an initial segmentation probability map;

[0066] A bidirectional mask calibration module, connected to the encoder-decoder network module, is used to reconstruct an error-corrected and semantically aligned ROI attention map in the feature space using the initial segmentation probability map as a priori.

[0067] The ROI feature extraction module, connected to the bidirectional mask calibration module, is used to extract multi-granularity ROI feature maps based on the ROI attention map.

[0068] The discriminative sampling attention module is connected to the ROI feature extraction module and is used to generate enhanced ROI feature maps and spatial feature weight maps of discriminative features corresponding to each granularity based on multi-granularity ROI feature maps.

[0069] A multi-granularity fusion module, connected to the discriminative sampling attention module, is used to fuse the enhanced ROI feature maps of each to obtain a fused feature vector;

[0070] The classification module, connected to the multi-granularity fusion module, is used to output the classification results of the image based on the fused feature vector. Attached Figure Description

[0071] Figure 1 This is a flowchart of the application process. Detailed Implementation

[0072] First, those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the embodiments of this application and are not intended to limit the scope of protection of the embodiments of this application. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0073] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application based on the specific circumstances.

[0074] In the embodiments of this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0075] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0076] Example 1:

[0077] See Figure 1 As shown in the figure, this application discloses a segmentation-guided image classification method based on a hierarchical attention mechanism, including:

[0078] Step 1: Obtain the image to be processed. In this embodiment of the application, the image is a medical image.

[0079] Step 2: Use an encoder-decoder network to extract features from the input image to obtain multi-level feature maps. Based on the deepest feature map Decode the data to generate an initial segmentation probability map. ;

[0080] Specifically, the encoder consists of multiple sequentially connected coding blocks. Each coding block contains a convolutional layer and a downsampling layer. The input medical image is processed by each coding block. The shallow coding blocks extract and output high-resolution feature maps containing rich detail information. However, feature maps The semantics are relatively weak; deep coding blocks extract and output low-resolution feature maps containing high-level semantic information. However, feature maps A great deal of detail and spatial location information was lost;

[0081] The decoder in the encoder-decoder network is coupled to the encoder and receives the deepest feature map output by the encoder. The spatial resolution of the feature map is gradually restored through a series of upsampling operations and skip connections. Finally, a convolutional layer followed by an activation function is used to map the number of channels to match the number of target classes, outputting an initial segmentation probability map. The initial segmentation probability map The numerical value at each spatial location represents the predicted probability that the pixel belongs to a specific category; this provides spatial prior guidance for subsequent steps; the size of the initial segmentation probability map is... , Indicates the number of channels. Indicates altitude, Indicates the width.

[0082] Step 3: Based on the initial segmentation probability map, a bidirectional mask calibration method is used to reconstruct an error-corrected and semantically aligned ROI attention map in the feature space; the initial segmentation probability map Including foreground probability diagram and background probability map Specifically, it includes:

[0083] Step 301: Calculate the background feature map from the background probability map using element-wise multiplication. The foreground feature map is calculated from the foreground probability map. Specifically, it includes:

[0084] The initial segmentation probability map output is a 2-channel probability map. ,in, Indicates the confidence level of the prospect. Indicates background confidence level; ;

[0085] Foreground Feature Map Calculation: First, adjust the foreground confidence level. Up to multi-level feature maps The spatial dimensions are determined, and regions with confidence values ​​greater than 0.5 are considered as coarse foreground regions. Then, element-wise multiplication is used to transform the multi-level feature maps. Features belonging to the foreground region are preserved, while low-probability region features are suppressed. The formula is expressed as:

[0086] ;

[0087] In the formula, Indicates characteristic functions; This represents element-wise multiplication;

[0088] Background feature map Calculation: First, adjust the background confidence level. Up to multi-level feature maps The spatial size is determined, and then regions with confidence values ​​greater than 0.5 are considered as coarse background regions. Element-wise multiplication is then used to refine the multi-level feature maps. Features belonging to the background region are preserved, while low-probability region features are suppressed. The formula is expressed as:

[0089] .

[0090] Step 302, from the background feature map In this process, high-confidence background feature vectors are sampled, and the semantic similarity between each background feature vector and the foreground feature map is calculated to obtain the response map of the background false detection region to be suppressed; specifically including:

[0091] First, a lightweight, learnable multilayer bandpass resampler is used to measure the background confidence. Adjust to Size; expression:

[0092] ;

[0093] In the formula, Indicates the base resampler. Represents a nonlinear function. This represents the learnable scaling factor and bias. This indicates that the filter bank is constructed by iterative application of the same underlying Gaussian filter, expressed as: , , , Represents the identity matrix;

[0094] Secondly, the background confidence after resampling Select the top with the highest confidence level The position of one pixel; Dynamically set to the number of background pixels in the initial segmentation probability map Embodiments of this application It ranges from 15% to 30%;

[0095] Next, based on the previous The position of each pixel is obtained from the background feature map. Background feature vector obtained by mid-sampling ;

[0096] Then, a binary gating function G is used to filter out potential foreground features, thereby obtaining the filtered background feature vector. The calculation expression is as follows:

[0097] :

[0098] ;

[0099] ;

[0100] Finally, for each background feature vector , and foreground feature map Calculate the cosine similarity to obtain A similarity graph, denoted as . , Subsequently, the background false detection region response map is obtained by performing position-by-position maximum fusion on the similarity maps. The calculation expression is:

[0101] ;

[0102] The corresponding feature map elements are semantically similar to the representative background features, therefore, The weights in the equation naturally represent the responses that should be suppressed within the prospect confidence level.

[0103] Step 303: Sample high-confidence foreground feature vectors from the foreground feature map, calculate the semantic similarity between each foreground feature vector and the background feature map, apply a Gaussian distance decay term to each semantic similarity, and obtain the response map of the foreground missed detection region to be recovered.

[0104] First, a lightweight, learnable multilayer bandpass resampler is used to measure the foreground confidence. Adjust to Size; expression:

[0105] ;

[0106] Secondly, from the foreground confidence after resampling Select the top with the highest confidence level The position of one pixel; Dynamically set to the number of foreground pixels in the initial segmentation probability map Embodiments of this application It ranges from 15% to 30%;

[0107] Next, based on the previous The position of each pixel is obtained from the foreground feature map. Foreground feature vector obtained by mid-sampling ;

[0108] Then, a binary gating function G is used to filter out potential background features, thereby obtaining the filtered foreground feature vector. :

[0109] ;

[0110] ;

[0111] Finally, for each foreground feature vector , and background feature map Calculate the cosine similarity to obtain A similarity graph, denoted as . Subsequently, considering the spatial clustering of ROIs in medical images, this embodiment applies a Gaussian distance attenuation term to each cosine similarity to suppress high similarity responses that may originate from distant background features, thereby obtaining a foreground missed detection region response map of the ignored foreground region. The calculation expression is:

[0112] ;

[0113] In the formula, and They represent , coordinates This indicates the intensity of the spatial distance penalty; for stable optimization, it will... Reparameterization ,in, Indicates a learnable scalar. This represents a small constant.

[0114] Step 304: Fuse the foreground probability map, the foreground missed detection region response map, and the background false detection region response map to generate the ROI attention map. The expression is:

[0115] ;

[0116] ROI Attention Map Reconstruction in the feature space effectively suppresses background-biased activation and restores foreground-biased activation, thus serving as a more refined and semantically aligned localization map.

[0117] Step 4, based on the ROI attention map Spatially weight the multi-level feature maps to extract multi-granularity ROI feature maps. ;

[0118] Step 5: Based on the discriminative sampling attention mechanism, generate enhanced ROI feature maps and spatial feature weight maps of discriminative features corresponding to each granularity on the multi-granularity ROI feature maps; specifically including:

[0119] Step 501: The multi-granularity ROI feature map outputs probabilistic sampling maps of each granularity through linear mapping and the Softmax function; the calculation expression is:

[0120] ;

[0121] In the formula, Indicates the sampling rate. Indicates the first One ROI feature map.

[0122] Step 502: Based on the probability weights of each channel in the sampling images of each granularity, perform a weighted summation of all spatial locations in the multi-granularity ROI feature map to obtain the discriminative feature descriptor; the calculation expression is:

[0123] ;

[0124] In the formula, Indicates the first A discriminative feature descriptor, Indicates spatial location Above, belonging to the first The probability of a discriminative feature; ROI feature map The size is , express The number of channels, express Spatial dimension.

[0125] Step 503: Apply a random discard operation to the discriminative feature descriptors to obtain the filtered discriminative feature descriptors; the calculation expression is:

[0126] ;

[0127] In the formula, This represents the discriminative feature descriptor after filtering. Indicates the discard rate. , Indicates reservation A discriminative feature descriptor.

[0128] Step 504: The highest-granularity ROI feature map is processed by a learnable weight matrix. Mapped to query space The selected discriminative feature descriptors are then processed using a learnable weight matrix. , Mapped to bond space respectively Value space The calculation expression is:

[0129] ;

[0130] ;

[0131] ;

[0132] Step 505, calculate the query space Bond space The similarity is used to obtain the attention weight matrix; the calculation expression is:

[0133] ;

[0134] In the formula, Represents the attention weight matrix; This indicates the position code.

[0135] Step 506, based on the attention weight matrix, adjust the value space. We perform a weighted summation to obtain the high-level ROI feature map enhanced by the current granularity discriminative features; the calculation expression is:

[0136] ;

[0137] In the formula, This represents the high-level ROI feature map.

[0138] Step 507: Perform global average pooling compression on each discriminative feature descriptor to obtain a set of scalars;

[0139] All scalars are input into the FFN network and activated by the Sigmoid activation function to obtain the importance score of each discriminative feature; the calculation expression is:

[0140] ;

[0141] In the formula, Indicates importance score, Used to The channel is compressed to 1. The network consists of two fully connected layers and an intermediate normalization layer. This represents the activation function.

[0142] Step 508: Perform column-by-column Softmax normalization on each column of the attention weight matrix; and combine the importance score with the weighted sum of the normalized attention columns to obtain the spatial feature weight map used to highlight the discrimination region; the calculation expression is:

[0143] ;

[0144] In the formula, Represents the spatial feature weight map. Represents the column index of the attention weight matrix.

[0145] Step 509: The high-level ROI feature map and the spatial feature weight map are further fused by weighted addition. After adjusting the dimensions using the projection matrix of multi-head attention, the enhanced ROI feature map is obtained. The calculation expression is:

[0146] ;

[0147] In the formula, This represents an enhanced ROI feature map. The projection matrix representing multi-head attention. This represents the learnable scaling factor.

[0148] Step 6: Using a multi-granularity weighted fusion strategy, aggregate the spatial feature weight maps and fuse the enhanced ROI feature maps to obtain a fused feature vector; specifically including:

[0149] Step 601: Concatenate the enhanced ROI feature maps generated at each granularity to form a multi-granularity feature map. The calculation expression is:

[0150] ;

[0151] Existing methods typically use Global Average Pooling (GAP) for feature fusion before final classification, which treats each element equally, thus weakening the model's focus on discriminative regions. In contrast, the embodiments of this application employ Global Weighted Pooling (GWP) as the final feature fusion strategy to better emphasize discriminative regions.

[0152] Step 602: Weight the spatial feature weight maps generated at each granularity to form a comprehensive feature weight map. The calculation expression is:

[0153] ;

[0154] In the formula, This represents the learnable weight coefficients.

[0155] Step 603: Based on the comprehensive feature weight map, perform a global weighted pooling operation on the multi-granularity feature maps to obtain the fused feature vector. The calculation expression is:

[0156] ;

[0157] Step 7: Perform image classification based on the fused feature vector.

[0158] Example 2:

[0159] A segmentation-guided image segmentation system, comprising:

[0160] The image acquisition module is used to acquire the image to be processed.

[0161] The encoder-decoder network module, connected to the image acquisition module, is used to extract multi-level feature maps of the image and generate an initial segmentation probability map;

[0162] A bidirectional mask calibration module, connected to the encoder-decoder network module, is used to reconstruct an error-corrected and semantically aligned ROI attention map in the feature space using the initial segmentation probability map as a priori.

[0163] The ROI feature extraction module, connected to the bidirectional mask calibration module, is used to extract multi-granularity ROI feature maps based on the ROI attention map.

[0164] The discriminative sampling attention module is connected to the ROI feature extraction module and is used to generate enhanced ROI feature maps and spatial feature weight maps of discriminative features corresponding to each granularity based on multi-granularity ROI feature maps.

[0165] A multi-granularity fusion module, connected to the discriminative sampling attention module, is used to fuse the enhanced ROI feature maps of each to obtain a fused feature vector;

[0166] The classification module, connected to the multi-granularity fusion module, is used to output the classification results of the image based on the fused feature vector.

[0167] After constructing the above segmentation-guided image segmentation system, before performing the image classification operations in steps 1 to 7 of Embodiment 1, the segmentation-guided image segmentation system needs to be trained.

[0168] The loss function during training consists of segmentation loss. Classification loss and feature weight supervision loss The weighted summation is expressed as follows:

[0169] ;

[0170] In the formula, Dice loss is used as the supervision loss for the encoder-decoder network for segmentation.

[0171] Classification loss In the process of fusing feature vectors The classifier uses LayerNorm and a fully connected layer for final classification, employing the cross-entropy loss function. Optimize;

[0172] Feature weight supervised loss It is in fusing feature vectors Depends on the learned comprehensive feature weight map To ensure more accurate and robust feature weights, multi-granularity feature maps... The Grad-CAM is calculated, and the obtained Grad-CAM map is used to... Comprehensive feature weight map learned by distance loss supervision The calculation expression is:

[0173] ;

[0174] ;

[0175] ;

[0176] In the formula, Representing multi-granularity feature maps The number of channels, Indicates the first Predicting logit for the class; Represents partial derivatives, This represents the activation function.

[0177] In the description of the embodiments of this application, it should be noted that the terms "inner" and "outer" and other terms indicating direction or positional relationship are based on the direction or positional relationship shown in the drawings. This is only for the convenience of description and does not indicate or imply that the device or component must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this application.

[0178] In the description of this application, the references to terms such as "an embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0179] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A segmentation-guided image classification method based on a hierarchical attention mechanism, characterized in that, include: Step 1: Obtain the image to be processed; Step 2: Use an encoder-decoder network to extract features from the input image to obtain multi-level feature maps; Decoding is performed based on the deepest feature map to generate an initial segmentation probability map; Step 3: Based on the initial segmentation probability map, a bidirectional mask calibration method is used to reconstruct an error-corrected and semantically aligned ROI attention map in the feature space; Step 4: Based on the ROI attention map, spatial weighting is performed on the multi-level feature maps to extract multi-granularity ROI feature maps; Step 5: Based on the discriminative sampling attention mechanism, generate enhanced ROI feature maps and spatial feature weight maps of discriminative features corresponding to each granularity on the multi-granularity ROI feature maps. Step 6: Through a multi-granularity weighted fusion strategy, aggregate the spatial feature weight maps and fuse the enhanced ROI feature maps to obtain a fused feature vector; Step 7: Perform image classification based on the fused feature vector.

2. The segmentation-guided image classification method based on hierarchical attention mechanism according to claim 1, characterized in that, The initial segmentation probability map includes a foreground probability map and a background probability map. Step 3 specifically includes: Step 301: Element-wise multiplication is used to calculate the background feature map from the background probability map and the foreground feature map from the foreground probability map. Step 302: Sample high-confidence background feature vectors from the background feature map, and calculate the semantic similarity between each background feature vector and the foreground feature map to obtain the response map of the background false detection region to be suppressed; Step 303: Sample high-confidence foreground feature vectors from the foreground feature map, calculate the semantic similarity between each foreground feature vector and the background feature map, apply a Gaussian distance decay term to each semantic similarity, and obtain the response map of the foreground missed detection region to be recovered. Step 304: The foreground probability map, the foreground missed detection region response map, and the background false detection region response map are fused to generate the ROI attention map.

3. The segmentation-guided image classification method based on hierarchical attention mechanism according to claim 2, characterized in that, In step 302, when sampling high-confidence background feature vectors, and in step 303, when sampling high-confidence foreground feature vectors, a lightweight, learnable multi-layer bandpass resampler is used to resample the initial segmentation probability map. After sampling high-confidence background feature vectors and high-confidence foreground feature vectors, the sampled background feature vectors and foreground feature vectors are filtered by a binary gating function.

4. The segmentation-guided image classification method based on hierarchical attention mechanism according to claim 3, characterized in that, The high-confidence background feature vectors specifically include: Select the highest confidence level from the background probability map of the initial segmentation probability map after resampling. 1 pixel; Based on selection The position of each pixel is located from the background feature map, and the corresponding background feature vector is sampled. Dynamically set to the number of background pixels in the initial segmentation probability map ; The sampling of high-confidence foreground feature vectors specifically includes: In the foreground probability map of the initial segmentation probability map after resampling, select the foreground with the highest confidence. 1 pixel; Based on selection The position of each pixel is located from the foreground feature map, and the corresponding foreground feature vector is sampled. Dynamically set to the number of foreground pixels in the initial segmentation probability map .

5. The segmentation-guided image classification method based on hierarchical attention mechanism according to claim 1, characterized in that, Step 5 specifically includes: Step 501: The multi-granularity ROI feature map outputs probabilistic sampling maps of each granularity through linear mapping and the Softmax function; Step 502: Based on the probability weights of each channel in the sampling map of each granularity, perform a weighted summation of all spatial locations in the multi-granularity ROI feature map to obtain the discriminative feature descriptor. Step 503: Apply a random discard operation to the discriminative feature descriptors to obtain the filtered discriminative feature descriptors; Step 504: The highest-granularity ROI feature map is processed by a learnable weight matrix. Mapped to query space The selected discriminative feature descriptors are then processed using a learnable weight matrix. , Mapped to bond space respectively Value space ; Step 505, calculate the query space Bond space The similarity is used to obtain the attention weight matrix; Step 506, based on the attention weight matrix, adjust the value space. Perform a weighted summation to obtain a high-level ROI feature map enhanced by the current granularity discriminative features; Step 507: Perform global average pooling compression on the selected discriminative feature descriptors to obtain a set of scalars; input all scalars into the FFN network and activate them through the Sigmoid activation function to obtain the importance score of each discriminative feature; Step 508: Perform column-by-column Softmax normalization on each column of the attention weight matrix; and combine the importance score to perform a weighted summation on the normalized attention columns to obtain a spatial feature weight map for highlighting the discrimination region. Step 509: The high-level ROI feature map and the spatial feature weight map are further fused by weighted addition. After adjusting the dimensions through the projection matrix of multi-head attention, an enhanced ROI feature map is obtained.

6. The segmentation-guided image classification method based on hierarchical attention mechanism according to claim 5, characterized in that, The calculation expression for step 505 is: ； In the formula, Represents the attention weight matrix; Indicates position code, Indicates reservation A discriminative feature descriptor, Indicates the first The number of channels in a ROI feature map. Indicates matrix transpose. This represents the height and width of the 4th ROI feature map; Step 506 calculates the expression as follows: ； In the formula, This represents the high-level ROI feature map.

7. The segmentation-guided image classification method based on hierarchical attention mechanism according to claim 6, characterized in that, The calculation expression for step 507 is: ； In the formula, Indicates importance score, This represents the discriminative feature descriptor after filtering. Used to The channel is compressed to 1. The network consists of two fully connected layers and an intermediate normalization layer. Indicates the activation function; The calculation expression for step 509 is: ； In the formula, This represents an enhanced ROI feature map. Represents the spatial feature weight map. The projection matrix representing multi-head attention. This represents the learnable scaling factor.

8. The segmentation-guided image classification method based on hierarchical attention mechanism according to claim 1, characterized in that, Step 6 specifically includes: Step 601: Concatenate the enhanced ROI feature maps generated at each granularity to form a multi-granularity feature map; Step 602: Weight the spatial feature weight maps generated at each granularity to form a comprehensive feature weight map; Step 603: Based on the comprehensive feature weight map, perform a global weighted pooling operation on the multi-granularity feature map to obtain the fused feature vector.

9. A segmentation-guided image segmentation system for performing the segmentation-guided image classification method based on a hierarchical attention mechanism as described in any one of claims 1-8, characterized in that, include: The image acquisition module is used to acquire the image to be processed. The encoder-decoder network module, connected to the image acquisition module, is used to extract multi-level feature maps of the image and generate an initial segmentation probability map; A bidirectional mask calibration module, connected to the encoder-decoder network module, is used to reconstruct an error-corrected and semantically aligned ROI attention map in the feature space using the initial segmentation probability map as a priori. The ROI feature extraction module, connected to the bidirectional mask calibration module, is used to extract multi-granularity ROI feature maps based on the ROI attention map. The discriminative sampling attention module is connected to the ROI feature extraction module and is used to generate enhanced ROI feature maps and spatial feature weight maps of discriminative features corresponding to each granularity based on multi-granularity ROI feature maps. A multi-granularity fusion module, connected to the discriminative sampling attention module, is used to fuse the enhanced ROI feature maps of each to obtain a fused feature vector; The classification module, connected to the multi-granularity fusion module, is used to output the classification results of the image based on the fused feature vector.

Images