Feature relation refinement-based weak supervision aerial image semantic segmentation method

Abstract

The invention discloses a feature relation refinement-based weak supervision aerial image semantic segmentation method, which comprises the following steps of: extracting image features of an aerial image and text features of a corresponding category through a pre-trained CLIP network, and generating an initial class activation graph (CAM) based on a classification score; then, a feature decoupling decoder is constructed to decouple the image features, and decoupling features and a preliminary segmentation prediction map are generated; a sparse selection attention refining module is constructed, and the CAM is dynamically corrected in combination with decoupling features; and finally, generating a high-quality pseudo tag and a feature relationship tag, and using the feature relationship tag as a supervision signal to realize optimization training of the weak supervision aerial image semantic segmentation network. According to the method, a mutual supervision mechanism of the decoupling decoder and the sparse selection attention refinement module is utilized, dynamic optimization of pseudo labels is achieved, the problems that an existing weak supervision semantic segmentation network is poor in target scale difference adaptability and poor in cross-domain generalization performance are solved, and the segmentation precision of a complex scene is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a weakly supervised aerial image semantic segmentation method based on feature relationship refinement. Background Art

[0002] In recent years, with the rapid development of drone platforms and sensor technology, low-altitude drone aerial imagery has been widely used in urban planning, agricultural management, ecological monitoring, military reconnaissance, and other fields. However, efficient analysis of aerial imagery data faces significant challenges: traditional fully supervised semantic segmentation methods rely on large amounts of pixel-level annotated data to drive model training. However, low-altitude aerial imagery, due to its wide coverage, high spatial resolution, diverse object categories, and highly heterogeneous distribution, is extremely expensive to label. Therefore, weakly supervised semantic segmentation has become an important research direction, significantly reducing the low-labeling burden by leveraging low-cost supervisory signals.

[0003] While weakly supervised semantic segmentation has made significant progress in recent years, due to the unique bird's-eye view and complex scene characteristics of aerial images, weakly supervised semantic segmentation methods for natural images are difficult to directly transfer to aerial imagery. Furthermore, current mainstream methods for generating initial pseudo-labels based on class activation maps (CAMs) also have inherent limitations: CAMs tend to be highly responsive to local discriminative regions, resulting in incomplete pseudo-label coverage or inclusion of background noise; and convolutional neural networks with fixed receptive fields struggle to adapt to the extreme scale variations in aerial imagery, leading to missed detection of small objects or disjointed segmentation of large objects. Summary of the Invention

[0004] The purpose of the present invention is to provide a weakly supervised semantic segmentation method for aerial images based on feature relationship refinement, which fully utilizes the cross-modal feature representation capability of CLIP, and constructs a feature decoupling decoder and a sparse selective attention refinement module, thereby solving the problems of poor adaptability to target scale differences and weak cross-domain generalization capability existing in previous weakly supervised semantic segmentation networks.

[0005] The technical solution to achieve the purpose of the present invention is: a weakly supervised aerial image semantic segmentation method based on feature relationship refinement, comprising the following steps:

[0006] Step S1: Crop and clean the images in the public dataset Aeroscapes to expand the dataset. 5154 images with a pixel size of 512×512 are obtained to form the training set, and 1266 images with a pixel size of 512×512 are obtained to form the validation set. Then proceed to step S2.

[0007] Step S2: Construct a weakly supervised aerial image semantic segmentation network based on feature relationship refinement. The weakly supervised aerial image semantic segmentation network based on feature relationship refinement uses the pre-trained CLIP as the backbone network, including an image encoder, a text encoder, a feature decoupling decoder, and a sparse selective attention refinement module, and then proceeds to step S3.

[0008] Step S3: Use the training set to train a weakly supervised aerial image semantic segmentation network based on feature relationship refinement to obtain a trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement, as follows:

[0009] Step S3-1: Input the images of the training set into the image encoder to extract image features; at the same time, input the image class labels of the training set into the text encoder to extract text features, and use the extracted image features and their corresponding text features to generate an initial CAM.

[0010] Step S3-2: Input the image features into the feature decoupling decoder to generate decoupled features, and use the decoupled features to further generate an initial segmentation prediction map.

[0011] Step S3-3: Input the image features into the sparse selective attention refinement module, use the decoupled features to construct feature relationships, and use the feature relationships as a guide to perform sparse selective attention operations on the image features extracted by the image encoder to obtain attention features with high confidence.

[0012] Step S3-4: Use the attention features obtained in step S3-3 and the initial CAM obtained in step S3-1 to refine the initial segmentation prediction map obtained in step S3-2, convert the refined segmentation prediction map into feature relationship labels, and use the feature relationship labels as supervision signals to train the feature decoupling decoder to update the decoupling features, and then update the feature relationship and refine the segmentation prediction map, finally obtaining a trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement, and proceed to step S4.

[0013] Step S4: Input the validation set into the trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement to obtain semantic segmentation results and evaluation results to evaluate the accuracy of the trained model.

[0014] Compared with the prior art, the advantages of the present invention are:

[0015] (1) A weakly supervised semantic segmentation method for aerial images based on feature relationship refinement with pre-trained CLIP as the backbone network is proposed. The cross-modal representation capability of the CLIP model is utilized to convert category label text into prior knowledge to guide the segmentation task, thereby enhancing the cross-domain adaptability of the model.

[0016] (2) A feature decoupling decoder based on the Transformer architecture was constructed for segmentation prediction, and a sparse selective attention refinement module was designed. The decoupled features generated by the feature decoupling decoder were used to dynamically correct the static CAM image, enabling the model to more accurately capture target areas of different scales and solving the problem of poor adaptability to target scale differences.

[0017] (3) Feature relationship labels are generated using the dynamically updated CAM, and the feature relationship labels are iteratively refined through the mutual supervision mechanism of the feature decoupling decoder and the sparse selection attention refinement module, providing the network with more accurate and effective supervision signals and further optimizing the network segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flowchart of the semantic segmentation of weakly supervised aerial images based on feature relationship refinement of the present invention.

[0019] Figure 2 This is a network structure diagram of the semantic segmentation of weakly supervised aerial images based on feature relationship refinement in the present invention. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0021] Combine Figure 1 and Figure 2 ,The weakly supervised aerial image semantic segmentation method based on feature relationship refinement,,including the following steps:

[0022] Step S1: Crop and clean the images in the public dataset Aeroscapes to expand the dataset. 5154 images with a pixel size of 512×512 are obtained to form the training set, and 1266 images with a pixel size of 512×512 are obtained to form the validation set. Then proceed to step S2.

[0023] Step S2: Construct a weakly supervised aerial image semantic segmentation network based on feature relationship refinement. The weakly supervised aerial image semantic segmentation network based on feature relationship refinement uses the pre-trained CLIP as the backbone network, including an image encoder, a text encoder, a feature decoupling decoder, and a sparse selective attention refinement module, and then proceeds to step S3.

[0024] Step S3: Use the training set to train a weakly supervised aerial image semantic segmentation network based on feature relationship refinement to obtain a trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement, as follows:

[0025] Step S3-1: Input the training set images into the image encoder to extract image features; at the same time, input the training set image class labels into the text encoder to extract text features, and use the extracted image features and their corresponding text features to generate the initial CAM, as follows:

[0026] Considering the computational resource limitations that may be faced by UAV scenarios, a pre-trained CLIP network with ViT-B as the backbone is used in the training process. Using the pre-trained CLIP network with ViT-B as the backbone, the training set images are input to the CLIP image encoder, and the initial image features are generated from the output of each Transformer block of the image encoder. Then we get the initial feature set l represents the index of the Transformer block, and N represents the number of Transformer blocks. At the same time, the foreground and background class labels of the training set images are used to construct text prompts and input into the text encoder to generate corresponding text features. Then, the image features and their corresponding text features are pooled, the distance between the two after pooling is calculated, and the softmax activation function is used to obtain the classification score. Based on the classification score, the Grad-CAM method is used to generate the initial CAM, which is denoted as C I Indicates the number of categories contained in the image, H indicates the image height, and W indicates the image width.

[0027] Step S3-2: Input the image features into the feature decoupling decoder to generate decoupled features, and use the decoupled features to further generate an initial segmentation prediction map, as follows:

[0028] Aerial images have high spatial resolution and diverse target scales. Premature cross-channel feature blending will cause small target features to be submerged. In order to retain image detail features, the initial feature set is first Each initial image feature Input to the depth mapping module DW-MLP to obtain the transition feature set

[0029]

[0030] Where, and represents two different fully connected layers, dwconv(·) represents depthwise convolution, and GeLU(·) represents the GeLU activation function.

[0031] Then, The transition features in are spliced together and sent to the convolution layer for feature fusion to obtain the fusion feature F fuTo avoid excessive smoothing of the fused features, the fused features are decoupled by separating the high-frequency and low-frequency features and then re-weighted to obtain the decoupled features F. de :

[0032] F de =α(avg pooling (F fu ))+(1-α)(F fu -inter(avg pooling (F fu )))

[0033] Where, avg pooling represents average pooling, inter represents interpolation, and α represents the hyperparameter of the balance weight.

[0034] Finally, the initial segmentation prediction map P is generated through several multi-head Transformer layers:

[0035] P=upsample(conv(Φ(F de ))

[0036] Where conv represents the convolutional layer, upsample represents upsampling, and Φ represents the multi-head Transformer layer. Each Transformer layer includes a multi-head self-attention module, a feedforward network, and two regularization layers.

[0037] Step S3-3: Input the image features into the sparse selective attention refinement module, use the decoupled features to build feature relationships, and use the feature relationships as a guide to perform sparse selective attention operations on the image features extracted by the image encoder to obtain high-confidence attention features, as follows:

[0038] In order to retain the useful prior knowledge in CLIP and remove the noise relationship, the decoupled feature F generated in step S3-2 is used de Establishing feature relationships

[0039] R de =sigmoid(F de T F de ),

[0040] In the formula, Compressed into two dimensions Further calculations are performed, where d, h, and w represent the channel size, height, and width of the feature map, sigmoid(·) represents the sigmoid(·) activation function to ensure that the output value range is (0, 1), and T represents the matrix transpose.

[0041] The initial feature set obtained in step S3-1 Each initial image feature in Perform sparse selective attention operation:

[0042] The initial image features are normalized by layer normalization, 1×1 convolution, and 3×3 depth convolution. Channel context encoding is performed to apply attention in the channel dimension; after encoding, the encoded feature map is obtained, and then the encoded feature map is divided into a query matrix, a key matrix, and a value matrix. The similarity between each pair of pixels in the reshaped query and key matrices is calculated to obtain a pixel-level query-key pair matrix, and the pixel-level query-key pair matrix is transposed to obtain the final query-key pair matrix M; unlike the dropout strategy that randomly discards scores, the top-k contribution scores of M are adaptively selected to retain the most important parts and delete useless scores; where k is an adjustable parameter for dynamically controlling the sparsity value, that is, only the top k values in each row of M that are within the range are sent to softmax for normalization, and for other elements less than the top-k contribution scores, the scatter function is used to replace their values with 0 at the given index:

[0043]

[0044] Where, Indicates sparsity k m The attention matrix at this time, Q, K, V represent the matrix forms of query, key, and value respectively. represents the scaling factor, which is used to control the sharpness of the attention distribution. represents the learnable top-k selection operation, k m represents the sparsity in the top-k selection strategy; i represents the i-th row, j represents the j-th column; M i,j Indicates the elements in each row that have a contribution score greater than top-k.

[0045] By adjusting the k value, multiple M with different sparsity levels are taken for attention calculation, and their weighted sum is used as the final attention matrix:

[0046]

[0047] Among them, A STKA Represents the final attention matrix after weighted summation of attention matrices with different sparsity, k m ={128, 170, 192, 204}, C Q represents the number of channels of the query vector matrix, ω m Represents trainable parameters corresponding to different sparsity, which are used to dynamically adjust the attention weights of different sparsity.

[0048] Finally, the initial feature set All The final attention matrix obtained after the above sparse selective attention operation is concatenated along the channel dimension, and the high-confidence attention features are obtained through linear projection.

[0049] Step S3-4: Use the attention features obtained in step S3-3 and the initial CAM obtained in step S3-1 to refine the initial segmentation prediction map obtained in step S3-2, convert the refined segmentation prediction map into feature relationship labels, and use the feature relationship labels as supervision signals to train the feature decoupling decoder to update the decoupling features, and then update the feature relationship and refine the segmentation prediction map, and finally obtain a trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement, as follows:

[0050] The final attention matrix A obtained in step S3-3 STKA and feature relationship R de Combined to construct the refined feature relationship R and refined CAM:

[0051] R=R de ×A STKA

[0052]

[0053] In the formula, c represents the category, represents the refined CAM for category c, R nor Indicates normalization of the rows and columns of R, θ represents the hyperparameter, Denotes the initial CAM for category c.

[0054] Then build a new R de Learning process: Use refined CAM to generate the final refined segmentation prediction map M P , and then M P Convert to feature relationship labels

[0055] Feature relationship labels The expression is:

[0056]

[0057] Where O(·) represents one-hot encoding;

[0058] The constructed loss function loss is as follows:

[0059]

[0060] Where, L cerepresents the cross entropy loss function, and β is the weight factor.

[0061] Then the feature relationship label The feature decoupling decoder is trained as a supervisory signal to update the decoupled features, and then update the feature relationship and refine the segmentation prediction map, ultimately obtaining a trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement.

[0062] use More accurate feature relationships can be established for pseudo-labels, resulting in higher-quality segmentation prediction maps, which can in turn be used to construct more precise feature relationships. The interaction between the feature decoupling decoder and the sparse selective attention refinement module is reused to promote network training, ultimately obtaining a trained weakly supervised aerial image semantic segmentation network based on feature relationship refinement.

[0063] Step S4: Input the validation set into the trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement, obtain the semantic segmentation result map and evaluation results, and evaluate the accuracy of the trained model.

[0064] Example 1

[0065] The weakly supervised aerial image semantic segmentation method based on feature relationship refinement described in the present invention comprises the following steps:

[0066] Step S1: Crop and clean the images in the public dataset Aeroscapes to expand the dataset. 5154 images with a pixel size of 512×512 are obtained to form the training set, and 1266 images with a pixel size of 512×512 are obtained to form the validation set. Then proceed to step S2.

[0067] Step S2: Construct a weakly supervised aerial image semantic segmentation network based on feature relationship refinement. The weakly supervised aerial image semantic segmentation network based on feature relationship refinement uses the pre-trained CLIP as the backbone network, including an image encoder, a text encoder, a feature decoupling decoder, and a sparse selective attention refinement module, and then proceeds to step S3.

[0068] Step S3: Use the training set to train a weakly supervised aerial image semantic segmentation network based on feature relationship refinement to obtain a trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement, as follows:

[0069] Step S3-1: Input the images of the training set into the image encoder to extract image features; at the same time, input the image class labels of the training set into the text encoder to extract text features, and use the extracted image features and their corresponding text features to generate an initial CAM.

[0070] Step S3-2: Input the image features into the feature decoupling decoder to generate decoupled features, and use the decoupled features to further generate an initial segmentation prediction map.

[0071] Step S3-3: Input the image features into the sparse selective attention refinement module, use the decoupled features to construct feature relationships, and use the feature relationships as a guide to perform sparse selective attention operations on the image features extracted by the image encoder to obtain attention features with high confidence.

[0072] Step S3-4: Use the attention features obtained in step S3-3 and the initial CAM obtained in step S3-1 to refine the initial segmentation prediction map obtained in step S3-2, convert the refined segmentation prediction map into feature relationship labels, and use the feature relationship labels as supervision signals to train the feature decoupling decoder to update the decoupling features, and then update the feature relationship and refine the segmentation prediction map, finally obtaining a trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement, and proceed to step S4.

[0073] Step S4: Input the validation set into the trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement to obtain semantic segmentation results and evaluation results to evaluate the accuracy of the trained model.

[0074] The method of the present invention uses Python programming language and PyTorch framework language to build a network framework and conduct relevant experiments on a computer equipped with a CPU of 12th Gen Intel(R) Core(TM) i9-12900K and a GPU of NVIDIAGeForce RTX 3090Ti. The method of the present invention uses the pre-trained CLIP with ViT-16-base architecture as the backbone network, sets the batch size to 4, N is a fixed number of 12, and the maximum number of iterations is 30,000. We use AdamW as the optimizer, with a weight decay of 1e-3, and initialize the learning rate to 2e-3. During the training process, all images are cropped to 320×320, and the weight factor β in the loss function is set to 0.1. Multiple trainings are conducted on the training set to obtain a weakly supervised aerial image semantic segmentation network based on feature relationship refinement.

[0075] To demonstrate the effectiveness of the present invention, we selected some mainstream weakly supervised semantic segmentation algorithms as comparison models. The comparative experimental results are shown in Table 1. The mIoU and F1 score indicators of the models are evaluated on the Aeroscapes dataset:

[0076] Table 1 Comparative experimental results of the present invention and other weakly supervised semantic segmentation algorithms on the Aeroscapes dataset

[0077]

[0078] Compared to state-of-the-art models like ToCo, FSR, and WeCLIP, our method achieves improvements in mIoU of 22.3%, 25.6%, and 5.8%, respectively. In terms of F1 score, our method also achieves significant improvements of 9.4%, 16%, and 2.4%, respectively, compared to other methods. Experimental results demonstrate the practicality and effectiveness of our method.

Claims

1. A weakly supervised aerial image semantic segmentation method based on feature relationship refinement, characterized in that: The following steps are involved: Step S1: Crop and clean the images in the public Aeroscapes dataset to expand the dataset. 5154 images with a pixel size of 512 × 512 are obtained to form the training set, and 1266 images with a pixel size of 512 × 512 are obtained to form the validation set. Then, proceed to step S2. Step S2: Construct a weakly supervised aerial image semantic segmentation network based on feature relationship refinement. The weakly supervised aerial image semantic segmentation network based on feature relationship refinement uses the pre-trained CLIP as the backbone network, including an image encoder, a text encoder, a feature decoupling decoder, and a sparse selective attention refinement module, and then proceeds to step S3. Step S3: Use the training set to train a weakly supervised aerial image semantic segmentation network based on feature relationship refinement to obtain a trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement, as follows: Step S3-1: Input the training set images into the image encoder to extract image features; at the same time, input the training set image class labels into the text encoder to extract text features, and use the extracted image features and their corresponding text features to generate an initial CAM; Step S3-2: input the image features into the feature decoupling decoder to generate decoupled features, and use the decoupled features to further generate an initial segmentation prediction map; Step S3-3: Input the image features into the sparse selective attention refinement module, use the decoupled features to construct feature relationships, and use the feature relationships as a guide to perform sparse selective attention operations on the image features extracted by the image encoder to obtain high-confidence attention features; Step S3-4: Using the attention features obtained in step S3-3 and the initial CAM obtained in step S3-1, refine the initial segmentation prediction map obtained in step S3-2, convert the refined segmentation prediction map into feature relationship labels, and use the feature relationship labels as supervision signals to train the feature decoupling decoder to update the decoupling features, and then update the feature relationship and refine the segmentation prediction map, finally obtaining a trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement, and then proceeding to step S4; Step S4: Input the validation set into the trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement to obtain semantic segmentation results and evaluation results to evaluate the accuracy of the trained model.

2. The weakly supervised aerial image semantic segmentation method based on feature relationship refinement according to claim 1 is characterized in that: In step S3-1, the images of the training set are input into the image encoder to extract image features; at the same time, the image class labels of the training set are input into the text encoder to extract text features, and the extracted image features and their corresponding text features are used to generate the initial CAM, as follows: Using the pre-trained CLIP network with ViT-B as the backbone, the training set images are input to the CLIP image encoder, and the initial image features are generated from the output of each Transformer block of the image encoder. Then we get the initial feature set l represents the index of the Transformer block, and N represents the number of Transformer blocks. At the same time, the foreground and background class labels of the training set images are used to construct text prompts and input into the text encoder to generate corresponding text features. Then, the image features and their corresponding text features are pooled, the distance between the two after pooling is calculated, and the softmax activation function is used to obtain the classification score. Based on the classification score, the Grad-CAM method is used to generate the initial CAM, which is denoted as C I Indicates the number of categories contained in the image, H indicates the image height, and W indicates the image width.

3. The weakly supervised aerial image semantic segmentation method based on feature relationship refinement according to claim 1, characterized in that: In step S3-2, the image features are input into the feature decoupling decoder to generate decoupled features, which are then used to generate an initial segmentation prediction map, as follows: First, the initial feature set Each initial image feature Input to the depth mapping module DW-MLP to obtain the transition feature set Where, and represents two different fully connected layers, dwconv(·) represents depthwise convolution, and GeLU(·) represents the GeLU activation function; Then, Transition feature splicing in They are sent together into the convolutional layer for feature fusion to obtain the fusion feature f fu ; Use high-frequency and low-frequency feature separation extraction to decouple the fusion features and re-weight them to obtain the decoupled features F de : F de =α(avg pooling (F fu ))+(1-α)(f fu -inter(avg pooling (F fu ))) Where, avg pooling represents average pooling, inter represents interpolation, and α represents the hyperparameter of the balance weight; Finally, the initial segmentation prediction map P is generated through several multi-head Transformer layers: P=upsample(conv(Φ(F de ))) Where conv represents the convolutional layer, upsample represents upsampling, and Φ represents the multi-head Transformer layer. Each Transformer layer includes a multi-head self-attention module, a feedforward network, and two regularization layers.

4. The weakly supervised aerial image semantic segmentation method based on feature relationship refinement according to claim 1, characterized in that: In step S3-3, the image features are input into the sparse selective attention refinement module, and the feature relationship is constructed using the decoupled features. The sparse selective attention operation is performed on the image features extracted by the image encoder guided by the feature relationship to obtain the attention features with high confidence, as follows: Using the decoupling feature F generated in step S3-2 de Establishing feature relationships R de =sigmoid(F de T F de ), In the formula, Compressed into two dimensions Further calculations are performed, where d, h, and w represent the channel size, height, and width of the feature map, sigmoid(·) represents the sigmoid(·) activation function to ensure that the output value range is (0, 1), and T represents the matrix transpose; The initial feature set obtained in step S3-1 Each initial image feature in Perform sparse selective attention operation: The initial image features are normalized by layer normalization, 1×1 convolution, and 3×3 depth convolution. Channel context encoding is performed to apply attention in the channel dimension; After encoding, the encoded feature map is obtained, which is then split into a query matrix, a key matrix, and a value matrix. The similarity between each pair of pixels in the reshaped query and key matrices is calculated to obtain a pixel-level query-key pair matrix. The pixel-level query-key pair matrix is then transposed to obtain the final query-key pair matrix M. Unlike the dropout strategy that randomly discards scores, the top-k contribution scores of M are adaptively selected to retain the most important parts and remove useless scores. Among them, k is an adjustable parameter used to dynamically control the value of sparsity, that is, only the top k values in each row of M that are within the range will be sent to softmax for normalization. For other elements with less than the top-k contribution score, the scatter function is used to replace their values with 0 at the given index: Where, Indicates sparsity k m The attention matrix at this time, Q, K, V represent the matrix forms of query, key, and value respectively. represents the scaling factor, which is used to control the sharpness of the attention distribution. represents the learnable top-k selection operation, k m represents the sparsity in the top-k selection strategy; i represents the i-th row, j represents the j-th column; M i,j Indicates the elements in each row whose contribution scores are greater than top-k; By adjusting the k value, multiple M with different sparsity levels are taken for attention calculation, and their weighted sum is used as the final attention matrix: Among them, A STKA Represents the final attention matrix after weighted summation of attention matrices with different sparsity, k m ={C Q / 2,2C Q / 3,3C Q / 4,4C Q / 5},C Q represents the number of channels of the query vector matrix, ω m Represents trainable parameters corresponding to different sparsity, which are used to dynamically adjust the attention weights of different sparsity; Finally, the initial feature set All The final attention matrix obtained after the above sparse selective attention operation is concatenated along the channel dimension, and the high-confidence attention features are obtained through linear projection.

5. The weakly supervised aerial image semantic segmentation method based on feature relationship refinement according to claim 1, characterized in that: In step S3-4, the initial segmentation prediction map obtained in step S3-2 is refined using the attention features obtained in step S3-3 and the initial CAM obtained in step S3-1. The refined segmentation prediction map is converted into feature relationship labels, and the feature relationship labels are used as supervisory signals to train the feature decoupling decoder to update the decoupling features, and then update the feature relationship and refine the segmentation prediction map. Finally, a trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement is obtained, as follows: The final attention matrix A obtained in step S3-3 STKA and feature relationship R de Combined to construct the refined feature relationship R and refined CAM: R=R de ×A STKA In the formula, c represents the category, represents the refined CAM for category c, R nor Indicates normalization of the rows and columns of R, θ represents the hyperparameter, represents the initial CAM for category c; Then build a new R de Learning process: Use refined CAM to generate the final refined segmentation prediction map M P , and then M P Convert to feature relationship labels Feature relationship labels The expression is: Where O(·) represents one-hot encoding; The constructed loss function loss is as follows: Where, L ce represents the cross entropy loss function, β is the weight factor; Then the feature relationship label The feature decoupling decoder is trained as a supervisory signal to update the decoupled features, and then update the feature relationship and refine the segmentation prediction map, finally obtaining a trained weakly supervised aerial image semantic segmentation model based on feature relationship refinement.

Images