Remote sensing reference image segmentation method based on contrastive learning and cross attention

Abstract

The present invention discloses a remote sensing reference image segmentation method and device based on contrastive learning and cross-attention, comprising the following steps: obtaining a training sample set and performing fine-grained Gaussian noise enhancement on the training sample set; generating a positive and negative sample pair set based on the unenhanced and enhanced training sample sets; each training sample in the training sample set includes a remote sensing reference image for training and its corresponding text information; inputting the training sample set and the positive and negative sample pair set into an initial remote sensing reference image segmentation framework model for training; during the training process, constructing a total loss function consisting of a cross-entropy loss and a contrastive learning loss, wherein the training sample set is used to calculate the cross-entropy loss and the positive and negative sample pair set is used to calculate the contrastive learning loss; and inputting the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain a segmentation result. The present invention improves the accuracy and reliability of remote sensing reference image segmentation tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a remote sensing reference image segmentation method and device based on contrastive learning and cross attention. Background Art

[0002] Reference remote sensing image segmentation involves identifying and segmenting specific objects within a remote sensing image based on a given description. For example, in a given remote sensing image, the description might be "segment all farmland areas." The goal of the remote sensing reference image segmentation task is to use an algorithm to identify all portions of the remote sensing image that match the description of "farmland areas" and accurately segment them. This typically involves establishing a correspondence between the verbal description and the visual content, thereby accurately identifying and segmenting the objects specified in the description. This task is particularly common in the field of language and vision fusion research.

[0003] Tongji University disclosed a reference image segmentation method based on progressive visual features in its patent document "Reference Image Segmentation Method, Device and Storage Medium Based on Progressive Visual Features" with application number 2023117186399: This method segments the corresponding objects referred to by the text description in the image based on a given image and the corresponding text description. First, the multi-scale visual features and text features of the input image are extracted separately; then, using multi-cardinality learning, the visual features and text features of the same scale are fused separately to obtain multi-scale text-visual interaction features, and the detailed features of the reference image are perceived in detail and comprehensively; finally, a multi-scale context aggregation pyramid is used to obtain context-enhanced visual features at each scale, and the multi-scale context-enhanced visual features are fused to obtain the segmentation mask of the input image; this patent has the advantages of comprehensive and accurate semantic perception, reliable information fusion, and light resource consumption. The University of Electronic Science and Technology of China (UESTC) has published a reference image segmentation method based on cross-modal dual graph alignment in its patent application number 2022114452600, titled "A Reference Image Segmentation Method Based on Cross-Modal Dual Graph Alignment." This method, which belongs to the field of multimodal image segmentation, innovatively proposes a "part-unified whole" paradigm, mapping the extracted visual and textual features to a unified latent representation structure before cross-modal fusion. This patent facilitates the explicit alignment information extracted by the model, effectively enhancing the final segmentation effect.

[0004] However, current research on remote sensing reference image segmentation tasks is relatively limited, primarily focusing on experiments with a small number of backbone networks or single decoders. Furthermore, many existing methods focus on standard RGB images, and research on their performance with high-resolution remote sensing images remains inadequate, resulting in insufficient generalization capabilities for these existing methods. Furthermore, datasets often make insufficient use of information from positive and negative samples, preventing the distances between positive samples from being sufficiently close and the distances between negative samples from being effectively widened. Finally, existing methods are still insufficient in integrating visual and textual features, resulting in poor overall performance. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a remote sensing reference image segmentation method and device based on contrastive learning and cross attention. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a remote sensing reference image segmentation method based on contrastive learning and cross attention, the method comprising:

[0007] Acquire a training sample set, perform fine-grained Gaussian noise enhancement on the training sample set, and generate a positive and negative sample pair set based on the unenhanced training sample set and the enhanced training sample set; wherein each training sample in the training sample set includes a remote sensing reference image used for training and its corresponding text information;

[0008] Inputting the training sample set and the positive and negative sample pair set into an initial remote sensing reference image segmentation framework model for training to obtain a trained remote sensing reference image segmentation framework model; wherein, during the training process, a total loss function consisting of a cross entropy loss and a contrastive learning loss is constructed, the training sample set is used for calculating the cross entropy loss, and the positive and negative sample pair set is used for calculating the contrastive learning loss;

[0009] Input the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain the segmentation result;

[0010] The remote sensing reference image segmentation framework model includes a visual backbone feature extraction network, a text feature extraction network, a multi-head cross-attention module, a pixel-level attention module, a language gating module, and a lightweight mask decoder; wherein the visual backbone feature extraction network is used to perform multi-scale feature extraction on the remote sensing reference image to be segmented to obtain visual extraction features; the text feature extraction network is used to perform feature extraction on the text information corresponding to the remote sensing reference image to be segmented to obtain text extraction features; the multi-head cross-attention module is used to fuse the visual extraction features and the text extraction features using a multi-head cross-attention mechanism to obtain a first visual-text feature; the pixel-level attention module is used to further fuse the first visual-text feature using a single-head scaled dot product attention mechanism to obtain a second visual-text feature; the language gating module is used to obtain a third visual-text feature by controlling the flow of text features in the second visual-text feature in the model; the lightweight mask decoder is used to generate a pixel-level segmentation mask based on the second visual-text feature and the third visual-text feature, and obtain a segmentation result based on the pixel-level segmentation mask.

[0011] In one embodiment of the present invention, performing fine-grained Gaussian noise enhancement on the training sample set includes:

[0012] Fine-grained Gaussian noise enhancement is performed on the remote sensing reference images in the training sample set, and fine-grained Gaussian noise enhancement is not required for text information corresponding to the remote sensing reference images in the training sample set.

[0013] In one embodiment of the present invention, the visual backbone feature extraction network is constructed by one or more of ResNet, MiT, PVTv2, LSKNet, Swin Transformer, Swin Transformer V2, ConvNeXt, UniRepLKNet, InternImage, FocalNet, VMamba, Vision Transformer series backbone networks and their subdivided backbones.

[0014] In one embodiment of the present invention, the visual backbone feature extraction network includes 63 backbone networks, which are constructed by ResNet, MiT, PVTv2, LSKNet, Swin Transformer, Swin Transformer V2, ConvNeXt, UniRepLKNet, InternImage, FocalNet, VMamba, Vision Transformer series backbone networks and their subdivided backbones.

[0015] In one embodiment of the present invention, the text feature extraction network is implemented by a Bert encoder.

[0016] In one embodiment of the present invention, the multi-head cross attention module utilizes a multi-head cross attention mechanism to fuse the visual extraction features and the text extraction features to obtain a first visual-text feature, including:

[0017] Using the visual extraction feature map as a query to form a first query matrix, and using the text extraction feature map as a key and a value to form a first key matrix and a first value matrix;

[0018] Calculating an attention score for each attention head based on the visual extracted features, the text extracted features, the first query matrix, and the first key matrix;

[0019] Normalize the attention score of each attention head using the Softmax function, and calculate the weighted average of each attention head based on the first value matrix and the normalized attention score of each attention head;

[0020] The first visual-text feature is obtained by concatenating the weighted average of all attention heads and linearly transforming them.

[0021] In one embodiment of the present invention, the pixel-level attention module further fuses the first visual-text features using a single-head scaled dot product attention mechanism to obtain a second visual-text feature, including:

[0022] Aggregating the text features in the second visual-text features at each spatial position in a word-level dimension to form a sentence-level feature, and the sentence-level features at all spatial positions constitute a new text feature;

[0023] Using the visual feature mapping in the second visual-textual feature as a query to form a second query matrix, and using the new text feature mapping in the second visual-textual feature as a key and a value to form a second key matrix and a second value matrix;

[0024] Calculating a single-head attention score based on the visual features and the new textual features in the second visual-textual feature, the second query matrix, and the second key matrix;

[0025] Normalizing the single-head attention score using a Softmax function, and calculating a weighted average based on the second value matrix and the normalized single-head attention score;

[0026] The weighted average is linearly transformed to obtain a second visual-text feature.

[0027] In one embodiment of the present invention, the remote sensing reference image to be segmented and its corresponding text information, as well as the remote sensing reference image used for training and its corresponding text information, are all from the RefSegRS dataset or the RRSIS-D dataset.

[0028] In a second aspect, an embodiment of the present invention provides a remote sensing reference image segmentation device based on contrastive learning and cross attention, the device comprising:

[0029] A data generation module is configured to obtain a training sample set, perform fine-grained Gaussian noise enhancement on the training sample set, and generate a set of positive and negative sample pairs based on the unenhanced training sample set and the enhanced training sample set; wherein each training sample in the training sample set includes a remote sensing reference image used for training and its corresponding text information;

[0030] A training module is configured to input the training sample set and the positive-negative sample pair set into an initial remote sensing reference image segmentation framework model for training to obtain a trained remote sensing reference image segmentation framework model; wherein, during the training process, a total loss function consisting of a cross entropy loss and a contrastive learning loss is constructed, the training sample set is used to calculate the cross entropy loss, and the positive-negative sample pair set is used to calculate the contrastive learning loss;

[0031] The segmentation module is used to input the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain the segmentation result;

[0032] The remote sensing reference image segmentation framework model includes a visual backbone feature extraction network, a text feature extraction network, a multi-head cross-attention module, a pixel-level attention module, a language gating module, and a lightweight mask decoder; wherein the visual backbone feature extraction network is used to perform multi-scale feature extraction on the remote sensing reference image to be segmented to obtain visual extraction features; the text feature extraction network is used to perform feature extraction on the text information corresponding to the remote sensing reference image to be segmented to obtain text extraction features; the multi-head cross-attention module is used to fuse the visual extraction features and the text extraction features using a multi-head cross-attention mechanism to obtain a first visual-text feature; the pixel-level attention module is used to further fuse the first visual-text feature using a single-head scaled dot product attention mechanism to obtain a second visual-text feature; the language gating module is used to obtain a third visual-text feature by controlling the flow of text features in the second visual-text feature in the model; the lightweight mask decoder is used to generate a pixel-level segmentation mask based on the second visual-text feature and the third visual-text feature, and obtain a segmentation result based on the pixel-level segmentation mask.

[0033] Beneficial effects of the present invention:

[0034] The remote sensing reference image segmentation method based on contrastive learning and cross-attention proposed in the present invention is specially designed for the segmentation of reference images in the remote sensing field. The method combines the existing remote sensing reference image dataset and effectively expands the remote sensing reference image dataset by performing fine-grained Gaussian noise enhancement on the existing remote sensing reference image dataset, thereby improving the generalization ability of the model. In addition, the expanded remote sensing reference image dataset generates a set of positive and negative sample pairs, which are used for contrastive learning training to calculate the contrastive learning loss and use it as part of the total training loss. This method enhances the model's ability to distinguish features and robustness. Furthermore, in the stage of fusion of visual features and text features, the embodiment of the present invention introduces a multi-head cross-attention module, a pixel-level attention module and language gating, which further improves the depth and accuracy of feature fusion. By using a multi-head cross-attention mechanism and a single-head scaling attention mechanism, the complex relationship between images and text can be better captured, thereby improving the accuracy and reliability of image segmentation tasks. In general, the embodiments of the present invention comprehensively consider the supplementation of training sample sets, the design of loss functions, and the design of models during the training process. It not only exhibits better performance when processing remote sensing reference images, but also has stronger adaptability and higher effect stability, making it suitable for processing high-resolution visible light remote sensing images collected by satellites or aircraft to achieve accurate segmentation of land object categories under language guidance.

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 1 is a flow chart of a remote sensing reference image segmentation method based on contrastive learning and cross attention provided by an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of the structure of the remote sensing reference image segmentation framework model provided by an embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of the structure of a pixel-level attention module provided by an embodiment of the present invention;

[0039] Figure 4 This is a structural diagram of a remote sensing reference image segmentation device based on contrastive learning and cross attention provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0041] First, see Figure 1The embodiment of the present invention provides a remote sensing reference image segmentation method based on contrastive learning and cross attention, which specifically includes the following steps:

[0042] S10, obtaining a training sample set, and performing fine-grained Gaussian noise enhancement on the training sample set, generating a positive and negative sample pair set based on the unenhanced training sample set and the enhanced training sample set; wherein each training sample in the training sample set includes a remote sensing reference image used for training and its corresponding text information.

[0043] The training sample set obtained in the embodiment of the present invention is from the RefSegRS dataset or the RRSIS-D dataset; Gaussian noise is added to the RefSegRS dataset or the RRSIS-D dataset to achieve a fine-grained data enhancement effect, thereby generating an enhanced RefSegRS_Aug dataset and an enhanced RRSIS-D_Aug dataset, respectively. More preferably, fine-grained Gaussian noise enhancement is performed on the training sample set, including: fine-grained Gaussian noise enhancement is performed on the remote sensing reference images in the training sample set, and fine-grained Gaussian noise enhancement is not required for the text information corresponding to the remote sensing reference images in the training sample set.

[0044] Taking the RefSegRS dataset as an example, assuming the number of samples in the RefSegRS dataset is N and the batch size is set to 2, the number of samples in the dataset after fine-grained Gaussian noise enhancement is 2N, which includes both RefSegRS and RefSegRS_Aug. For example, within a batch size, four samples are generated: remote sensing reference images I1 and I2 from the RefSegRS dataset, along with their corresponding text information T1 and T2; remote sensing reference images I1' and I2' from the RefSegRS_Aug dataset, along with their corresponding text information T1' and T2'. Since fine-grained Gaussian noise enhancement is performed only on the remote sensing reference images, T1 and T1' are equal, and T2 and T2' are equal. Finally, we get I1, I2, I1', I2', T1, and T2, where I1 and I1', I2 and I2' form two positive sample pairs, and I1 and I2, I1' and I2' form two negative sample pairs. That is, one batch size can get four positive and negative sample pairs, and the positive and negative sample pairs corresponding to all batch sizes constitute the positive and negative sample pair set.

[0045] The fine-grained data enhancement method of the embodiment of the present invention effectively expands the data set, improves the generalization ability of the model, and enables the model to better adapt to various noises and changes during the training process, thereby improving the final segmentation effect.

[0046] S20. Input the training sample set and the positive and negative sample pair set into the initial remote sensing reference image segmentation framework model for training to obtain a trained remote sensing reference image segmentation framework model; wherein, during the training process, a total loss function consisting of a cross entropy loss and a contrastive learning loss is constructed, the training sample set is used to calculate the cross entropy loss, and the positive and negative sample pair set is used to calculate the contrastive learning loss.

[0047] In order to achieve better segmentation effect, the embodiment of the present invention constructs a remote sensing reference image segmentation framework model such as Figure 2 As shown in the figure, the remote sensing reference image segmentation framework model includes a visual backbone feature extraction network, a text feature extraction network, a multi-head cross attention module, a pixel-level attention module, a language gating module, and a lightweight mask decoder; wherein,

[0048] The visual backbone feature extraction network is used to perform multi-scale feature extraction on the remote sensing reference image to be segmented to obtain visual extraction features.

[0049] In the optional solution provided by an embodiment of the present invention, the visual backbone feature extraction network is constructed by one or more of the ResNet, MiT, PVTv2, LSKNet, Swin Transformer, Swin Transformer V2, ConvNeXt, UniRepLKNet, InternImage, FocalNet, VMamba, Vision Transformer series backbone networks and their subdivided backbones.

[0050] In the optional solution provided by an embodiment of the present invention, the visual backbone feature extraction network includes 63 backbone networks, and the 63 backbone networks are constructed by ResNet, MiT, PVTv2, LSKNet, Swin Transformer, Swin Transformer V2, ConvNeXt, UniRepLKNet, InternImage, FocalNet, VMamba, Vision Transformer series backbone networks and their subdivided backbones.

[0051] In the above two optional schemes: ResNet series includes resnet50 and resnet101 subdivision backbone networks; MiT series includes mit_b0, mit_b1, mit_b2, mit_b3, mit_b4, and mit_b5 subdivision backbone networks; PVTv2 series includes pvtv2_b0, pvtv2_b1, pvtv2_b2, pvtv2_b3, pvtv2_b4, and pvtv2_b5 subdivision backbone networks; LSKNet series includes lsknet_tiny and lsknet_small subdivision backbone networks; Swin Transformer series includes swin_tiny, swin_small, swin_base, swin_large, and swin_base_gfm_w6 subdivision backbone networks; Swin The TransformerV2 series includes swinv2_tiny, swinv2_small, and swinv2_base subdivision backbone networks; the ConvNeXt series includes convnext_tiny, convnext_small, convnext_base, convnext_large, and convnext_xlarge subdivision backbone networks; the UniRepLKNet series includes unireplknet_tiny, unireplknet_small, unireplknet_base, unireplknet_large, and unireplknet_xlarge subdivision backbone networks; Inter The nImage series includes internimage_tiny, internimage_small, internimage_base, internimage_large, internimage_xlarge, and internimage_huge subdivided backbone networks; the FocalNet series includes focalnet_tiny, focalnet_small, focalnet_base, focalnet_large, and focalnet_xlarge subdivided backbone networks; the VMamba series includes vmamba_tiny, vmamba_small, and vmamba_base subdivided backbone networks;The VisionTransformer series includes vit_base, vit_large, vit_large_patch16_satmae_pp_timm, vit_large_patch16_satmae_pp_rsfm, vit_large_patch16_scalemae_timm, vit_large_patch16_scalemae_rsfm, vit_large_patch16_cross_scalemae_timm, vit_large_patch16_cross_scalemae_rsfm, vit_base_patch16_spectralgpt, vit_base_patch16_spatsigma, vit_large_patch16_spatsigma, vit_huge_patch16_spatsigma, vit_base_patch16_hypersigma, vit_large_patch16_hypersigma, vit_huge_patch16_hypersigma, a total of 63 backbone networks. It can be seen that the visual backbone feature extraction network of the embodiment of the present invention can be constructed by some or all of the subdivided backbone networks in these 63 backbone networks. The specific visual backbone feature extraction network can be selected and designed according to actual needs.

[0052] Finally, all I1, I2, I1', and I2' generated in S10 are input into the visual backbone feature extraction network to obtain multi-scale visual extraction features.

[0053] Furthermore, the text feature extraction network of the embodiment of the present invention is used to extract features from text information corresponding to the remote sensing reference image to be segmented, and obtain text extraction features.

[0054] The text feature extraction network in this embodiment of the present invention is implemented using an existing Bert encoder. The Bert encoder is composed of multiple stacked Transformer encoders, each of which consists of a multi-head self-attention and feedforward network. All samples T1 and T2 generated in S10 are input into the Bert encoder to obtain text extraction features.

[0055] Furthermore, the multi-head cross attention module of the embodiment of the present invention is used to use the multi-head cross attention mechanism to fuse the visual extraction features and the text extraction features to obtain the first visual-text feature.

[0056] In the multi-head cross attention module of the embodiment of the present invention, a multi-head cross attention mechanism is used to fuse the visual extraction features and the text extraction features to obtain the first visual-text features, including:

[0057] The visual extraction feature map is used as a query to form a first query matrix, and the text extraction feature map is used as a key and value to form a first key matrix and a first value matrix; the attention score of each attention head is calculated based on the visual extraction features and the text extraction features, as well as the first query matrix and the first key matrix; the attention score of each attention head is normalized using the Softmax function, and the weighted average of each attention head is calculated based on the first value matrix and the normalized attention score of each attention head; the weighted averages of all attention heads are spliced ​​and linearly transformed to obtain the first visual-text feature. More specifically:

[0058] This embodiment of the present invention calculates the attention score of each attention head, and the formula is expressed as:

[0059]

[0060] in, represents the attention score of the h-th attention head, Q h represents the first query matrix of the h-th attention head, K h represents the first bond matrix of the h-th attention head, d k represents the key dimension of the h-th attention head, and X represents the visual extraction features and text extraction features input into the multi-head cross attention module.

[0061] In this embodiment of the present invention, the attention score of each attention head is normalized using the Softmax function, and then the weighted average of each attention head is calculated. The formula is expressed as:

[0062]

[0063] in, represents the weighted average of the h-th attention head, represents the normalized attention score of the h-th attention head, V h Represents the first value matrix of the hth attention head. Finally, the weighted average of all attention heads is concatenated and linearly transformed, and the formula is expressed as:

[0064]

[0065] Among them, H represents the number of attention heads in the multi-head cross attention module, Concat represents the weighted average of all attention heads connected together, and W O Represents a linear transformation matrix, usually a weight matrix.

[0066] In an embodiment of the present invention, the visual extraction features output by the visual backbone feature extraction network and the text extraction features output by the text feature extraction network are input into a multi-head cross attention module, with the visual extraction feature map being used as a query and the text extraction feature map being used as a key and value to perform multi-head cross attention. After the visual extraction features and the text extraction features are processed by multi-head cross attention, the output at this time is element-wise multiplied by the input visual extraction features and linearly transformed and projected through a 1×1 convolutional layer. The projection includes an instance normalization layer, and finally a fused first visual-text feature is obtained, in which the visual features and text features have the same spatial size.

[0067] The embodiment of the present invention introduces a multi-head cross-attention mechanism in the fusion stage of visual features and text features. Through parallel calculation and fusion of multiple attention heads, the relationship between visual features and text features is further enhanced. The multi-head cross-attention mechanism not only improves the depth and accuracy of feature fusion, but also can better capture the complex relationship between images and text, thereby improving the accuracy and reliability of image segmentation tasks.

[0068] Furthermore, the pixel-level attention module of the embodiment of the present invention is used to further fuse the first visual-text features using a single-head scaled dot product attention mechanism to obtain a second visual-text feature.

[0069] In the pixel-level attention module of the embodiment of the present invention, the first visual-text features are further fused using a single-head scaled dot product attention mechanism to obtain a second visual-text feature, including:

[0070] The text features in the second visual-text feature are aggregated at the word level in each spatial position to form a sentence-level feature, and the sentence-level features of all spatial positions constitute a new text feature; the visual feature mapping in the second visual-text feature is used as a query to form a second query matrix, and the new text feature mapping in the second visual-text feature is used as a key and value to form a second key matrix and a second value matrix; the single-head attention score is calculated based on the visual features and new text features in the second visual-text feature, as well as the second query matrix and the second key matrix; the single-head attention score is normalized using the Softmax function, and a weighted average is calculated based on the second value matrix and the normalized single-head attention score; the weighted average is linearly transformed to obtain the second visual-text feature. More specifically:

[0071] The embodiment of the present invention provides a structure of a pixel-level attention module as follows Figure 3As shown, it includes the first to sixth convolutional layers and a Softmax layer. The second and fifth convolutional layers each contain an instance normalization layer. The visual features in the first visual-textual feature are input into the first and second convolutional layers, and the textual features in the first visual-textual feature are input into the third and fourth convolutional layers. The output of the second convolutional layer is multiplied by the output of the third convolutional layer and then input into the Softmax layer. The output of the Softmax layer is multiplied by the output of the fourth convolutional layer and then input into the fifth convolutional layer. The output of the first convolutional layer is element-wise multiplied by the output of the fifth convolutional layer and then input into the sixth convolutional layer. The output of the sixth convolutional layer is used as the second visual-textual feature. It can be seen that the pixel-level attention module takes the first visual-textual feature obtained by the multi-head cross-attention module as input, the visual feature map in the first visual-textual feature as the query, and the textual feature map in the first visual-textual feature as the key and value to perform scaled dot product attention. After performing scaled dot product attention on the first visual-textual feature, language feature aggregation is achieved, and a sentence-level feature vector at a specific position is generated at each spatial position to form a spatial feature map. Figure 3 The multiplication of the output of the Softmax layer and the output of the fourth convolutional layer is the spatial feature map. The spatial feature map is then projected using the fifth convolutional layer, which includes an instance normalization layer. The projected result of the fifth convolutional layer and the visual features in the first visual-textual feature after the first convolutional layer are multiplied element-wise using the scaled dot product attention formula to achieve multimodal fusion. The multiplication result is passed through the sixth convolutional layer to obtain the final fusion result of the pixel-level attention module, namely the second visual-textual feature. In the second visual-textual feature, the visual features and text features have the same spatial size. The first to sixth convolutional layers are all 1×1 convolutional layers.

[0072] The pixel-level attention module in this embodiment of the present invention performs pixel-level enhancement in the latent space of the image from the perspective of the image, and adds cross-attention in the feature space of the text as part of the enhancement, further strengthening the relationship between the two modalities of image and text features. In this way, text features can provide semantic supplements to image features, allowing the model to consider the contextual information in the text when processing the image, thereby improving overall comprehension and accuracy, improving the effect of cross-modal fusion, and achieving pixel-level enhancement of the image.

[0073] Furthermore, the language gating module of the embodiment of the present invention is used to obtain the third visual-textual feature by controlling the flow of text features in the second visual-textual feature in the model.

[0074] In an embodiment of the present invention, after passing through the pixel-level attention module, a language gating module is used to control the flow of text features in the second visual-text feature in the entire model. The language gating module includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence. The first convolutional layer and the second convolutional layer are 1×1 convolutional layers, the first activation layer is a RELU activation function, and the second activation layer is a TANH activation function. The output of the pixel-level attention module is dot-multiplied with the output of the second activation layer to obtain the third visual-text feature.

[0075] Furthermore, the lightweight mask decoder of the present invention is configured to generate a pixel-level segmentation mask based on the second and third visual-text features, and to obtain a segmentation result based on the pixel-level segmentation mask. Using an existing lightweight mask decoder, the present invention comprehensively considers the outputs of the pixel-level attention module and the language gating module, namely, the second and third visual-text features, to generate a pixel-level segmentation mask. This pixel-level segmentation mask can provide a more accurate segmentation result.

[0076] Furthermore, during the training process, the embodiment of the present invention constructs a total loss function consisting of cross entropy loss and contrastive learning loss, and adds contrastive learning loss as an additional loss term to the total loss. The total loss function formula is expressed as:

[0077] TotalLoss=SLoss+λ×CLoss;

[0078] Specifically, TotalLoss represents the total loss function, SLoss represents the main loss function used for remote sensing reference image segmentation tasks, such as cross-entropy loss, and λ represents the weight of the contrastive learning loss. In practical applications, λ is usually set to a small value, such as 0.1 to 0.5. CLoss represents the contrastive learning loss function. During training, the contrastive learning loss is optimized jointly with the main loss function for image segmentation.

[0079] Among them, the contrastive learning loss function is expressed as:

[0080]

[0081] Among them, N represents the number of training samples in the training sample set, x i represents the i-th training sample in the training sample set, x i + Represents the positive and negative sample pairs and the training sample x i The positive sample that constitutes the positive sample pair, f(x i ) represents x i After the output of the remote sensing reference image segmentation framework model, f(x i+ ) represents x i + After the output of the remote sensing reference image segmentation framework model, sim(f(xi),f(xi+)) means to find f(xi) and f(x i + ), K represents the number of negative samples in the positive and negative sample pairs, x j - represents the jth negative sample in the positive and negative sample set, f(x j - ) represents x j - After the output of the remote sensing reference image segmentation framework model, sim(f(xi),f(xj-)) represents the calculation of f(x) and f ( xj- ) The similarity between them is usually defined using cosine similarity, and τ represents the balance coefficient, which is used to adjust the smoothness of the distribution of positive and negative samples.

[0082] This embodiment of the present invention incorporates contrastive learning loss as an additional loss term into the overall loss function during training. By performing contrastive learning on an expanded dataset, the calculated contrastive learning loss is optimized together with the image segmentation loss function. This approach significantly enhances the model's ability to distinguish features and its robustness, improving its stability and accuracy when processing complex data.

[0083] S30 , inputting the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain a segmentation result.

[0084] The present embodiment first obtains a remote sensing reference image to be segmented and its corresponding text information from the RefSegRS dataset or the RRSIS-D dataset. The image and its corresponding text information are then fed into a trained remote sensing reference image segmentation framework model to obtain a segmentation result, which includes a ground object category label and a probability value for each pixel in the remote sensing reference image.

[0085] In summary, the embodiment of the present invention proposes a remote sensing reference image segmentation method based on contrastive learning and cross-attention, which is specially designed for the segmentation of reference images in the remote sensing field. The method combines the existing remote sensing reference image dataset and effectively expands the remote sensing reference image dataset by performing fine-grained Gaussian noise enhancement on the existing remote sensing reference image dataset, thereby improving the generalization ability of the model. In addition, the expanded remote sensing reference image dataset generates a set of positive and negative sample pairs, which are used for contrastive learning training, thereby calculating the contrastive learning loss and using it as part of the total training loss. This method enhances the model's ability to distinguish features and robustness. Furthermore, in the stage of fusing visual features with text features, the embodiment of the present invention introduces a multi-head cross-attention module, a pixel-level attention module and language gating, which further improves the depth and accuracy of feature fusion. By using a multi-head cross-attention mechanism and a single-head scaling attention mechanism, the complex relationship between images and text can be better captured, thereby improving the accuracy and reliability of image segmentation tasks. In general, the embodiments of the present invention comprehensively consider the supplementation of training sample sets, the design of loss functions, and the design of models during the training process. It not only exhibits better performance when processing remote sensing reference images, but also has stronger adaptability and higher effect stability, making it suitable for processing high-resolution visible light remote sensing images collected by satellites or aircraft to achieve accurate segmentation of land object categories under language guidance.

[0086] Second, see Figure 4 , an embodiment of the present invention provides a remote sensing reference image segmentation device based on contrastive learning and cross attention, the device comprising:

[0087] A data generation module is used to obtain a training sample set, perform fine-grained Gaussian noise enhancement on the training sample set, and generate a set of positive and negative sample pairs based on the unenhanced training sample set and the enhanced training sample set; wherein each training sample in the training sample set includes a remote sensing reference image used for training and its corresponding text information;

[0088] A training module is used to input the training sample set and the positive and negative sample pair set into the initial remote sensing reference image segmentation framework model for training, thereby obtaining a trained remote sensing reference image segmentation framework model. During the training process, a total loss function consisting of a cross entropy loss and a contrastive learning loss is constructed. The training sample set is used to calculate the cross entropy loss, and the positive and negative sample pair set is used to calculate the contrastive learning loss.

[0089] The segmentation module is used to input the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain the segmentation result; wherein,

[0090] The remote sensing reference image segmentation framework model includes a visual backbone feature extraction network, a text feature extraction network, a multi-head cross-attention module, a pixel-level attention module, a language gating module, and a lightweight mask decoder; among them, the visual backbone feature extraction network is used to perform multi-scale feature extraction on the remote sensing reference image to be segmented to obtain visual extraction features; the text feature extraction network is used to perform feature extraction on the text information corresponding to the remote sensing reference image to be segmented to obtain text extraction features; the multi-head cross-attention module is used to use the multi-head cross-attention mechanism to fuse the visual extraction features and the text extraction features to obtain the first visual-text feature; the pixel-level attention module is used to further fuse the first visual-text feature using the single-head scaled dot product attention mechanism to obtain the second visual-text feature; the language gating module is used to obtain the third visual-text feature by controlling the flow of text features in the second visual-text feature in the model; the lightweight mask decoder is used to generate a pixel-level segmentation mask based on the second visual-text feature and the third visual-text feature, and obtain the segmentation result based on the pixel-level segmentation mask.

[0091] As for the device embodiment of the second aspect, since it is basically similar to the method embodiment of the first aspect, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment of the first aspect.

[0092] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0093] Although the present invention is described herein in conjunction with various embodiments, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the specification and accompanying drawings in the process of implementing the claimed invention. In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components or steps. The fact that certain measures are described in different embodiments does not mean that these measures cannot be combined to produce good results.

[0094] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A remote sensing reference image segmentation method based on contrastive learning and cross attention, characterized in that The method comprises: Acquire a training sample set, perform fine-grained Gaussian noise enhancement on the training sample set, and generate a positive and negative sample pair set based on the unenhanced training sample set and the enhanced training sample set; wherein each training sample in the training sample set includes a remote sensing reference image used for training and its corresponding text information; Inputting the training sample set and the positive and negative sample pair set into an initial remote sensing reference image segmentation framework model for training to obtain a trained remote sensing reference image segmentation framework model; wherein, during the training process, a total loss function consisting of a cross entropy loss and a contrastive learning loss is constructed, the training sample set is used for calculating the cross entropy loss, and the positive and negative sample pair set is used for calculating the contrastive learning loss; Input the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain the segmentation result; The remote sensing reference image segmentation framework model includes a visual backbone feature extraction network, a text feature extraction network, a multi-head cross-attention module, a pixel-level attention module, a language gating module, and a lightweight mask decoder; wherein the visual backbone feature extraction network is used to perform multi-scale feature extraction on the remote sensing reference image to be segmented to obtain visual extraction features; the text feature extraction network is used to perform feature extraction on the text information corresponding to the remote sensing reference image to be segmented to obtain text extraction features; the multi-head cross-attention module is used to fuse the visual extraction features and the text extraction features using a multi-head cross-attention mechanism to obtain a first visual-text feature; the pixel-level attention module is used to further fuse the first visual-text feature using a single-head scaled dot product attention mechanism to obtain a second visual-text feature; the language gating module is used to obtain a third visual-text feature by controlling the flow of text features in the second visual-text feature in the model; the lightweight mask decoder is used to generate a pixel-level segmentation mask based on the second visual-text feature and the third visual-text feature, and obtain a segmentation result based on the pixel-level segmentation mask.

2. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that Performing fine-grained Gaussian noise enhancement on the training sample set, including: Fine-grained Gaussian noise enhancement is performed on the remote sensing reference images in the training sample set, and fine-grained Gaussian noise enhancement is not required for text information corresponding to the remote sensing reference images in the training sample set.

3. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that The visual backbone feature extraction network is constructed by one or more of the ResNet, MiT, PVTv2, LSKNet, Swin Transformer, Swin Transformer V2, ConvNeXt, UniRepLKNet, InternImage, FocalNet, VMamba, VisionTransformer series backbone networks and their subdivided backbones.

4. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that The visual backbone feature extraction network includes 63 backbone networks, which are constructed by ResNet, MiT, PVTv2, LSKNet, Swin Transformer, Swin Transformer V2, ConvNeXt, UniRepLKNet, InternImage, FocalNet, VMamba, Vision Transformer series backbone networks and their subdivided backbones.

5. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that The text feature extraction network is implemented by a Bert encoder.

6. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that The multi-head cross attention module utilizes a multi-head cross attention mechanism to fuse the visual extraction features and the text extraction features to obtain a first visual-text feature, including: Using the visual extraction feature map as a query to form a first query matrix, and using the text extraction feature map as a key and a value to form a first key matrix and a first value matrix; Calculating an attention score for each attention head based on the visual extracted features, the text extracted features, the first query matrix, and the first key matrix; Normalize the attention score of each attention head using the Softmax function, and calculate the weighted average of each attention head based on the first value matrix and the normalized attention score of each attention head; The first visual-text feature is obtained by concatenating the weighted average of all attention heads and linearly transforming them.

7. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that The pixel-level attention module further fuses the first visual-text features using a single-head scaled dot product attention mechanism to obtain a second visual-text feature, including: Aggregating the text features in the second visual-text features at each spatial position in a word-level dimension to form a sentence-level feature, and the sentence-level features at all spatial positions constitute a new text feature; Using the visual feature mapping in the second visual-textual feature as a query to form a second query matrix, and using the new text feature mapping in the second visual-textual feature as a key and a value to form a second key matrix and a second value matrix; Calculating a single-head attention score based on the visual features and the new textual features in the second visual-textual feature, the second query matrix, and the second key matrix; Normalizing the single-head attention score using a Softmax function, and calculating a weighted average based on the second value matrix and the normalized single-head attention score; The weighted average is linearly transformed to obtain a second visual-text feature.

8. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that The remote sensing reference images to be segmented and their corresponding text information, as well as the remote sensing reference images used for training and their corresponding text information, are all from the RefSegRS dataset or the RRSIS-D dataset.

9. A remote sensing reference image segmentation device based on contrastive learning and cross attention, characterized in that: The device comprises: A data generation module is configured to obtain a training sample set, perform fine-grained Gaussian noise enhancement on the training sample set, and generate a set of positive and negative sample pairs based on the unenhanced training sample set and the enhanced training sample set; wherein each training sample in the training sample set includes a remote sensing reference image used for training and its corresponding text information; A training module is configured to input the training sample set and the positive-negative sample pair set into an initial remote sensing reference image segmentation framework model for training to obtain a trained remote sensing reference image segmentation framework model; wherein, during the training process, a total loss function consisting of a cross entropy loss and a contrastive learning loss is constructed, the training sample set is used to calculate the cross entropy loss, and the positive-negative sample pair set is used to calculate the contrastive learning loss; The segmentation module is used to input the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain the segmentation result; The remote sensing reference image segmentation framework model includes a visual backbone feature extraction network, a text feature extraction network, a multi-head cross-attention module, a pixel-level attention module, a language gating module, and a lightweight mask decoder; wherein the visual backbone feature extraction network is used to perform multi-scale feature extraction on the remote sensing reference image to be segmented to obtain visual extraction features; the text feature extraction network is used to perform feature extraction on the text information corresponding to the remote sensing reference image to be segmented to obtain text extraction features; the multi-head cross-attention module is used to fuse the visual extraction features and the text extraction features using a multi-head cross-attention mechanism to obtain a first visual-text feature; the pixel-level attention module is used to further fuse the first visual-text feature using a single-head scaled dot product attention mechanism to obtain a second visual-text feature; the language gating module is used to obtain a third visual-text feature by controlling the flow of text features in the second visual-text feature in the model; the lightweight mask decoder is used to generate a pixel-level segmentation mask based on the second visual-text feature and the third visual-text feature, and obtain a segmentation result based on the pixel-level segmentation mask.

Images