Remote sensing reference image segmentation method based on comparative learning and cross attention

Abstract

The invention discloses a remote sensing reference image segmentation method and device based on comparative learning and cross attention, and the method comprises the steps: obtaining a training sample set, carrying out the fine-grained Gaussian noise enhancement of the training sample set, and generating a positive and negative sample pair set according to an unenhanced training sample set and an enhanced training sample set; each training sample in the training sample set comprises a remote sensing reference image for training and corresponding text information; inputting the training sample set and the positive and negative sample pair set into an initial remote sensing reference image segmentation framework model for training; in the training process, a total loss function composed of cross entropy loss and contrast learning loss is constructed, the training sample set is used for calculating the cross entropy loss, and the positive and negative sample pair set is used for calculating the contrast learning loss; and inputting the remote sensing reference image to be segmented and the text information corresponding to the remote sensing reference image to be segmented into the trained remote sensing reference image segmentation framework model to obtain a segmentation result. According to the invention, the accuracy and reliability of the remote sensing reference image segmentation task are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a remote sensing reference image segmentation method and device based on contrastive learning and cross attention. Background Art

[0002] Reference remote sensing image segmentation is to identify and segment specific objects in remote sensing images based on a given description. For example, in a given remote sensing image, the description may be "segment all farmland areas". At this time, the goal of the remote sensing reference image segmentation task is to use algorithms to identify all parts of the remote sensing image that meet the description of "farmland areas" and accurately segment them. This usually involves establishing a correspondence between the language description and the visual content, so as to accurately identify and segment the objects specified in the description. This task is usually more common in the field of language and visual fusion research.

[0003] Tongji University disclosed a reference image segmentation method based on progressive visual features in its patent document "Reference image segmentation method, device and storage medium based on progressive visual features" with application number 2023117186399: this method segments the corresponding objects referred to by the text description in the image according to the given image and the corresponding text description. First, the multi-scale visual features and text features of the input image are extracted respectively; then, multi-cardinality learning is used to fuse the visual features and text features of the same scale respectively, and multi-scale text-visual interaction features are obtained to perceive the detailed features of the reference image in detail and comprehensively; finally, a multi-scale context aggregation pyramid is used to obtain the context-enhanced visual features of each scale, and the multi-scale context-enhanced visual features are fused to obtain the segmentation mask of the input image; this patent has the advantages of comprehensive and accurate semantic perception, reliable information fusion and light resource consumption. The University of Electronic Science and Technology of China disclosed a reference image segmentation method based on cross-modal dual graph alignment in its patent document with application number 2022114452600, "A reference image segmentation method based on cross-modal dual graph alignment": This method belongs to the field of multimodal image segmentation; The patent innovatively proposed the "part-unified whole" paradigm, that is, the extracted visual and text features are first mapped to a unified potential representation structure, and then cross-modal fusion is performed. The patent is conducive to the explicit alignment information extracted by the model, which can effectively enhance the final segmentation effect.

[0004] However, firstly, the current research on the remote sensing reference image segmentation task is relatively limited, mainly focusing on experiments with a small number of backbone networks or a single decoder. At the same time, many existing methods focus on ordinary RGB images, and the performance research on high-resolution remote sensing images is still not in-depth enough, resulting in insufficient generalization ability of these existing methods. Secondly, the dataset is insufficient in the use of positive and negative sample information, so that the distance between positive samples is not sufficiently close, while the distance between negative samples is not effectively increased. Finally, the existing methods are still insufficient in the fusion of visual features and text features, resulting in poor overall performance. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a remote sensing reference image segmentation method and device based on contrastive learning and cross attention. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a remote sensing reference image segmentation method based on contrastive learning and cross attention, the method comprising:

[0007] Acquire a training sample set, perform fine-grained Gaussian noise enhancement on the training sample set, and generate a positive and negative sample pair set according to the unenhanced training sample set and the enhanced training sample set; wherein each training sample in the training sample set includes a remote sensing reference image used for training and its corresponding text information;

[0008] Inputting the training sample set and the positive and negative sample pair set into the initial remote sensing reference image segmentation framework model for training to obtain a trained remote sensing reference image segmentation framework model; wherein, during the training process, a total loss function consisting of a cross entropy loss and a contrastive learning loss is constructed, the training sample set is used for calculating the cross entropy loss, and the positive and negative sample pair set is used for calculating the contrastive learning loss;

[0009] Inputting the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain the segmentation result;

[0010] The remote sensing reference image segmentation framework model includes a visual backbone feature extraction network, a text feature extraction network, a multi-head cross-attention module, a pixel-level attention module, a language gating module, and a lightweight mask decoder; wherein the visual backbone feature extraction network is used to perform multi-scale feature extraction on the remote sensing reference image to be segmented to obtain visual extraction features; the text feature extraction network is used to perform feature extraction on the text information corresponding to the remote sensing reference image to be segmented to obtain text extraction features; the multi-head cross-attention module is used to fuse the visual extraction features and the text extraction features using a multi-head cross-attention mechanism to obtain a first visual-text feature; the pixel-level attention module is used to further fuse the first visual-text feature using a single-head scaled dot product attention mechanism to obtain a second visual-text feature; the language gating module is used to obtain a third visual-text feature by controlling the flow of text features in the second visual-text feature in the model; the lightweight mask decoder is used to generate a pixel-level segmentation mask according to the second visual-text feature and the third visual-text feature, and obtain a segmentation result according to the pixel-level segmentation mask.

[0011] In one embodiment of the present invention, fine-grained Gaussian noise enhancement is performed on the training sample set, including:

[0012] Fine-grained Gaussian noise enhancement is performed on the remote sensing reference images in the training sample set, and fine-grained Gaussian noise enhancement is not required for text information corresponding to the remote sensing reference images in the training sample set.

[0013] In one embodiment of the present invention, the visual backbone feature extraction network is constructed by one or more of ResNet, MiT, PVTv2, LSKNet, Swin Transformer, Swin Transformer V2, ConvNeXt, UniRepLKNet, InternImage, FocalNet, VMamba, Vision Transformer series backbone networks and their subdivided backbones.

[0014] In one embodiment of the present invention, the visual backbone feature extraction network includes 63 backbone networks, and the 63 backbone networks are constructed by ResNet, MiT, PVTv2, LSKNet, Swin Transformer, Swin Transformer V2, ConvNeXt, UniRepLKNet, InternImage, FocalNet, VMamba, Vision Transformer series backbone networks and their subdivided backbones.

[0015] In one embodiment of the present invention, the text feature extraction network is implemented by a Bert encoder.

[0016] In one embodiment of the present invention, the multi-head cross attention module uses a multi-head cross attention mechanism to fuse the visual extraction feature and the text extraction feature to obtain a first visual-text feature, including:

[0017] Using the visual extraction feature map as a query to form a first query matrix, and using the text extraction feature map as a key and a value to form a first key matrix and a first value matrix;

[0018] Calculating an attention score of each attention head according to the visual extraction features and the text extraction features, the first query matrix and the first key matrix;

[0019] Normalize the attention score of each attention head using the Softmax function, and calculate the weighted average of each attention head according to the first value matrix and the normalized attention score of each attention head;

[0020] The weighted average of all attention heads is concatenated and linearly transformed to obtain the first visual-text feature.

[0021] In one embodiment of the present invention, the pixel-level attention module further fuses the first visual-text feature using a single-head scaled dot product attention mechanism to obtain a second visual-text feature, including:

[0022] Aggregating the text features in the second visual-text features at each spatial position in a word-level dimension to form a sentence-level feature, and the sentence-level features at all spatial positions constitute a new text feature;

[0023] Using the visual feature mapping in the second visual-textual feature as a query to form a second query matrix, and using the new text feature mapping in the second visual-textual feature as a key and a value to form a second key matrix and a second value matrix;

[0024] Calculating a single-head attention score according to the visual features and the new text features in the second visual-text features, the second query matrix, and the second key matrix;

[0025] Normalizing the single-head attention score by using a Softmax function, and calculating a weighted average value according to the second value matrix and the normalized single-head attention score;

[0026] The weighted average is linearly transformed to obtain a second visual-text feature.

[0027] In one embodiment of the present invention, the remote sensing reference image to be segmented and its corresponding text information, as well as the remote sensing reference image used for training and its corresponding text information, are all from the RefSegRS dataset or the RRSIS-D dataset.

[0028] In a second aspect, an embodiment of the present invention provides a remote sensing reference image segmentation device based on contrastive learning and cross attention, the device comprising:

[0029] A data generation module, used to obtain a training sample set, perform fine-grained Gaussian noise enhancement on the training sample set, and generate a positive and negative sample pair set according to the unenhanced training sample set and the enhanced training sample set; wherein each training sample in the training sample set includes a remote sensing reference image used for training and its corresponding text information;

[0030] A training module, used for inputting the training sample set and the positive and negative sample pair set into the initial remote sensing reference image segmentation framework model for training, so as to obtain a trained remote sensing reference image segmentation framework model; wherein, during the training process, a total loss function consisting of a cross entropy loss and a contrastive learning loss is constructed, the training sample set is used for calculating the cross entropy loss, and the positive and negative sample pair set is used for calculating the contrastive learning loss;

[0031] A segmentation module is used to input the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain the segmentation result;

[0032] The remote sensing reference image segmentation framework model includes a visual backbone feature extraction network, a text feature extraction network, a multi-head cross-attention module, a pixel-level attention module, a language gating module, and a lightweight mask decoder; wherein the visual backbone feature extraction network is used to perform multi-scale feature extraction on the remote sensing reference image to be segmented to obtain visual extraction features; the text feature extraction network is used to perform feature extraction on the text information corresponding to the remote sensing reference image to be segmented to obtain text extraction features; the multi-head cross-attention module is used to fuse the visual extraction features and the text extraction features using a multi-head cross-attention mechanism to obtain a first visual-text feature; the pixel-level attention module is used to further fuse the first visual-text feature using a single-head scaled dot product attention mechanism to obtain a second visual-text feature; the language gating module is used to obtain a third visual-text feature by controlling the flow of text features in the second visual-text feature in the model; the lightweight mask decoder is used to generate a pixel-level segmentation mask according to the second visual-text feature and the third visual-text feature, and obtain a segmentation result according to the pixel-level segmentation mask.

[0033] Beneficial effects of the present invention:

[0034] The remote sensing reference image segmentation method based on contrastive learning and cross-attention proposed in the present invention is specially designed for the segmentation of reference images in the remote sensing field. The method combines the existing remote sensing reference image dataset and effectively expands the remote sensing reference image dataset by performing fine-grained Gaussian noise enhancement on the existing remote sensing reference image dataset, thereby improving the generalization ability of the model. In addition, the expanded remote sensing reference image dataset generates a set of positive and negative sample pairs, which are used for contrastive learning training, thereby calculating the contrastive learning loss and using it as part of the total training loss. This method enhances the model's ability to distinguish features and robustness. Furthermore, in the stage of fusion of visual features and text features, the embodiment of the present invention introduces a multi-head cross-attention module, a pixel-level attention module and language gating, which further improves the depth and accuracy of feature fusion. By using a multi-head cross-attention mechanism and a single-head scaling attention mechanism, the complex relationship between images and text can be better captured, thereby improving the accuracy and reliability of image segmentation tasks. In general, the embodiments of the present invention comprehensively consider the supplementation of training sample sets, the design of loss functions, and the design of models during the training process. It not only exhibits better performance when processing remote sensing reference images, but also has stronger adaptability and higher effect stability, making it suitable for processing high-resolution visible light remote sensing images collected by satellites or aircraft, so as to achieve accurate segmentation of land object categories under language guidance.

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of a remote sensing reference image segmentation method based on contrastive learning and cross attention provided by an embodiment of the present invention;

[0037] Figure 2 It is a structural schematic diagram of a remote sensing reference image segmentation framework model provided by an embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of the structure of a pixel-level attention module provided by an embodiment of the present invention;

[0039] Figure 4 It is a structural schematic diagram of a remote sensing reference image segmentation device based on contrastive learning and cross-attention provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0041] First, see Figure 1The embodiment of the present invention provides a remote sensing reference image segmentation method based on contrastive learning and cross attention, which specifically includes the following steps:

[0042] S10, obtaining a training sample set, and performing fine-grained Gaussian noise enhancement on the training sample set, generating a positive and negative sample pair set according to the unenhanced training sample set and the enhanced training sample set; wherein each training sample in the training sample set includes a remote sensing reference image used for training and its corresponding text information.

[0043] The training sample set obtained by the embodiment of the present invention is from the RefSegRS dataset or the RRSIS-D dataset; Gaussian noise is added to the RefSegRS dataset or the RRSIS-D dataset to achieve a fine-grained data enhancement effect, and an enhanced RefSegRS_Aug dataset and RRSIS-D_Aug dataset are generated respectively. More preferably, fine-grained Gaussian noise enhancement is performed on the training sample set, including: fine-grained Gaussian noise enhancement is performed on the remote sensing reference images in the training sample set, and fine-grained Gaussian noise enhancement is not required for the text information corresponding to the remote sensing reference images in the training sample set.

[0044] Taking the RefSegRS dataset as an example, assuming that the number of samples in the RefSegRS dataset is N, and the batchsize is set to 2, the number of samples in the dataset after fine-grained Gaussian noise enhancement is 2N, that is, it includes RefSegRS and RefSegRS_Aug. For example, four samples are generated in one batchsize, namely, the remote sensing reference images I1 and I2 from the RefSegRS dataset, and their corresponding text information T1 and T2, and the remote sensing reference images I1' and I2' from the RefSegRS_Aug dataset, and their corresponding text information T1' and T2'. Since fine-grained Gaussian noise enhancement is only performed on remote sensing reference images, T1 and T1' are equal, and T2 and T2' are equal. Finally, we get I1, I2, I1', I2', T1, and T2, where I1 and I1', I2 and I2' form two positive sample pairs, and I1 and I2, I1' and I2' form two negative sample pairs, that is, one batchsize can get four positive and negative sample pairs, and the positive and negative sample pairs corresponding to all batchsizes constitute the positive and negative sample pair set.

[0045] The fine-grained data enhancement method of the embodiment of the present invention effectively expands the data set, improves the generalization ability of the model, and enables the model to better adapt to various noises and changes during the training process, thereby improving the final segmentation effect.

[0046] S20. Input the training sample set and the positive and negative sample pair set into the initial remote sensing reference image segmentation framework model for training to obtain a trained remote sensing reference image segmentation framework model; wherein, during the training process, a total loss function consisting of a cross entropy loss and a contrastive learning loss is constructed, the training sample set is used to calculate the cross entropy loss, and the positive and negative sample pair set is used to calculate the contrastive learning loss.

[0047] In order to achieve better segmentation effect, the embodiment of the present invention constructs a remote sensing reference image segmentation framework model such as Figure 2 As shown in the figure, the remote sensing reference image segmentation framework model includes a visual backbone feature extraction network, a text feature extraction network, a multi-head cross attention module, a pixel-level attention module, a language gating module, and a lightweight mask decoder; wherein,

[0048] The visual backbone feature extraction network is used to perform multi-scale feature extraction on the remote sensing reference image to be segmented to obtain visual extraction features.

[0049] In the optional scheme provided by an embodiment of the present invention, the visual backbone feature extraction network is constructed by one or more of ResNet, MiT, PVTv2, LSKNet, Swin Transformer, Swin Transformer V2, ConvNeXt, UniRepLKNet, InternImage, FocalNet, VMamba, Vision Transformer series backbone networks and their subdivided backbones.

[0050] In the optional scheme provided by an embodiment of the present invention, the visual backbone feature extraction network includes 63 backbone networks, and the 63 backbone networks are constructed by ResNet, MiT, PVTv2, LSKNet, Swin Transformer, Swin Transformer V2, ConvNeXt, UniRepLKNet, InternImage, FocalNet, VMamba, Vision Transformer series backbone networks and their subdivided backbones.

[0051] In the above two optional schemes: ResNet series includes resnet50 and resnet101 subdivision backbone networks; MiT series includes mit_b0, mit_b1, mit_b2, mit_b3, mit_b4, and mit_b5 subdivision backbone networks; PVTv2 series includes pvtv2_b0, pvtv2_b1, pvtv2_b2, pvtv2_b3, pvtv2_b4, and pvtv2_b5 subdivision backbone networks; LSKNet series includes lsknet_tiny and lsknet_small subdivision backbone networks; Swin Transformer series includes swin_tiny, swin_small, swin_base, swin_large, and swin_base_gfm_w6 subdivision backbone networks; Swin The TransformerV2 series includes swinv2_tiny, swinv2_small, and swinv2_base subdivision backbone networks; the ConvNeXt series includes convnext_tiny, convnext_small, convnext_base, convnext_large, and convnext_xlarge subdivision backbone networks; the UniRepLKNet series includes unireplknet_tiny, unireplknet_small, unireplknet_base, unireplknet_large, and unireplknet_xlarge subdivision backbone networks; Inter The nImage series includes internimage_tiny, internimage_small, internimage_base, internimage_large, internimage_xlarge, and internimage_huge subdivided backbone networks; the FocalNet series includes focalnet_tiny, focalnet_small, focalnet_base, focalnet_large, and focalnet_xlarge subdivided backbone networks; the VMamba series includes vmamba_tiny, vmamba_small, and vmamba_base subdivided backbone networks;The VisionTransformer series includes vit_base, vit_large, vit_large_patch16_satmae_pp_timm, vit_large_patch16_satmae_pp_rsfm, vit_large_patch16_scalemae_timm, vit_large_patch16_scalemae_rsfm, vit_large_patch16_cross_scalemae_timm, vit_large_patch16_cross_scalemae_rsfm, vit_base_patch16_spectralgpt, vit_base_patch16_spatsigma, vit_large_patch16_spatsigma, vit_huge_patch16_spatsigma, vit_base_patch16_hypersigma, vit_large_patch16_hypersigma, vit_huge_patch16_hypersigma, a total of 63 backbone networks. It can be seen that the visual backbone feature extraction network of the embodiment of the present invention can be constructed by some or all of the subdivided backbone networks in these 63 backbone networks, and the specific visual backbone feature extraction network can be selected and designed according to actual needs. ;

[0052] Finally, all I1, I2, I1', and I2' generated in S10 are input into the visual backbone feature extraction network to obtain multi-scale visual extraction features.

[0053] Furthermore, the text feature extraction network of the embodiment of the present invention is used to extract features from text information corresponding to the remote sensing reference image to be segmented, and obtain text extraction features.

[0054] The text feature extraction network of the embodiment of the present invention is implemented by an existing Bert encoder. The Bert encoder is composed of multiple Transformer encoders stacked together, and each Transformer encoder layer is composed of a multi-head self-attention and feedforward network. All samples T1 and T2 generated in S10 are input into the Bert encoder to obtain text extraction features.

[0055] Furthermore, the multi-head cross attention module of the embodiment of the present invention is used to fuse the visual extraction features and the text extraction features using the multi-head cross attention mechanism to obtain the first visual-text feature.

[0056] In the multi-head cross attention module of the embodiment of the present invention, a multi-head cross attention mechanism is used to fuse the visual extraction features and the text extraction features to obtain the first visual-text features, including:

[0057] The visual extraction feature map is used as a query to form a first query matrix, and the text extraction feature map is used as a key and a value to form a first key matrix and a first value matrix; the attention score of each attention head is calculated based on the visual extraction feature and the text extraction feature, as well as the first query matrix and the first key matrix; the attention score of each attention head is normalized using the Softmax function, and the weighted average of each attention head is calculated based on the first value matrix and the normalized attention score of each attention head; the weighted averages of all attention heads are concatenated and linearly transformed to obtain the first visual-text feature. More specifically:

[0058] The embodiment of the present invention calculates the attention score of each attention head, and the formula is expressed as:

[0059]

[0060] in, represents the attention score of the h-th attention head, Q h represents the first query matrix of the h-th attention head, K h represents the first bond matrix of the h-th attention head, d k represents the key dimension of the hth attention head, and X represents the visual extraction features and text extraction features input into the multi-head cross attention module.

[0061] In this embodiment of the present invention, the attention score of each attention head is normalized using the Softmax function, and then the weighted average value of each attention head is calculated. The formula is:

[0062]

[0063] in, represents the weighted average of the h-th attention head, represents the normalized attention score of the hth attention head, V h Represents the first value matrix of the hth attention head. Finally, the weighted averages of all attention heads are concatenated and linearly transformed, and the formula is expressed as:

[0064]

[0065] Where H represents the number of attention heads in the multi-head cross attention module, Concat represents the weighted average of all attention heads connected together, and W O Represents a linear transformation matrix, usually a weight matrix.

[0066] The embodiment of the present invention inputs the visual extraction features output by the visual backbone feature extraction network and the text extraction features output by the text feature extraction network into a multi-head cross attention module, and uses the visual extraction feature map as a query and the text extraction feature map as a key and value to perform multi-head cross attention. After the visual extraction features and the text extraction features are processed by multi-head cross attention, the output at this time is element-wise multiplied by the input visual extraction features, and linear transformation projection is performed through a 1×1 convolutional layer, and an instance normalization layer is used in the projection, and finally a fused first visual-text feature is obtained, in which the visual features and text features have the same spatial size.

[0067] The embodiment of the present invention introduces a multi-head cross-attention mechanism in the fusion stage of visual features and text features. The multi-head cross-attention mechanism further enhances the relationship between visual features and text features through parallel calculation and fusion of multiple attention heads. The multi-head cross-attention mechanism not only improves the depth and accuracy of feature fusion, but also can better capture the complex relationship between images and texts, thereby improving the accuracy and reliability of image segmentation tasks.

[0068] Furthermore, the pixel-level attention module of the embodiment of the present invention is used to further fuse the first visual-text features using a single-head scaled dot product attention mechanism to obtain a second visual-text feature.

[0069] In the pixel-level attention module of the embodiment of the present invention, the first visual-text feature is further fused using a single-head scaled dot product attention mechanism to obtain a second visual-text feature, including:

[0070] The text features in the second visual-text features are aggregated at the word level in each spatial position to form sentence-level features, and the sentence-level features at all spatial positions constitute new text features; the visual feature mapping in the second visual-text features is used as a query to form a second query matrix, and the new text feature mapping in the second visual-text features is used as a key and a value to form a second key matrix and a second value matrix; the single-head attention score is calculated based on the visual features and the new text features in the second visual-text features, as well as the second query matrix and the second key matrix; the single-head attention score is normalized using the Softmax function, and a weighted average is calculated based on the second value matrix and the normalized single-head attention score; the weighted average is linearly transformed to obtain the second visual-text feature. More specifically:

[0071] The embodiment of the present invention provides a structure of a pixel-level attention module as follows Figure 3As shown, it includes the first to sixth convolutional layers, a Softmax layer, and the second and fifth convolutional layers each contain an instance normalization layer; wherein, the visual features in the first visual-textual feature are input into the first and second convolutional layers, the textual features in the first visual-textual feature are input into the third and fourth convolutional layers, the output of the second convolutional layer is multiplied by the output of the third convolutional layer and then input into the Softmax layer, the output of the Softmax layer is multiplied by the output of the fourth convolutional layer and then input into the fifth convolutional layer, the output of the first convolutional layer is element-wise multiplied by the output of the fifth convolutional layer and then input into the sixth convolutional layer, and the output of the sixth convolutional layer is used as the second visual-textual feature. It can be seen that the pixel-level attention module takes the first visual-textual feature obtained by the multi-head cross attention module as input, the visual feature map in the first visual-textual feature as a query, and the textual feature map in the first visual-textual feature as a key and value to perform scaled dot product attention. After the scaled dot product attention processing is performed on the first visual-textual feature, language feature aggregation is realized, and a sentence-level feature vector at a specific position is generated at each spatial position to form a spatial feature map. Figure 3 The multiplication result of the output of the Softmax layer in the 4th convolutional layer and the output of the 5th convolutional layer is the spatial feature map. After that, the fifth convolutional layer including the instance normalization layer is used to project the spatial feature map. The result after the projection of the fifth convolutional layer and the result after the projection of the visual features in the first visual-textual feature through the first convolutional layer are multiplied element by element using the scaled dot product attention formula to realize multimodal fusion. The multiplication result is passed through the sixth convolutional layer to obtain the final fusion result of the pixel-level attention module, i.e., the second visual-textual feature. The visual features and text features in the second visual-textual feature have the same spatial size. Among them, the first to sixth convolutional layers are all 1×1 convolutional layers.

[0072] The pixel-level attention module of the embodiment of the present invention performs pixel-level enhancement in the latent space of the image from the perspective of the image, and adds cross-attention in the feature space of the text as part of the enhancement, further strengthening the relationship between the two modalities of image and text features. In this way, text features can provide semantic supplements to image features, so that the model can take into account the contextual information in the text when processing images, thereby improving the overall understanding ability and accuracy, improving the effect of cross-modal fusion, and achieving pixel-level enhancement of the image.

[0073] Furthermore, the language gating module of the embodiment of the present invention is used to obtain the third visual-textual feature by controlling the flow of the textual feature in the second visual-textual feature in the model.

[0074] In the embodiment of the present invention, after passing through the pixel-level attention module, a language gating module is used to control the flow of text features in the second visual-text features in the entire model. The language gating module includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence. The first convolutional layer and the second convolutional layer are 1×1 convolutional layers, the first activation layer is a RELU activation function, and the second activation layer is a TANH activation function. The output of the pixel-level attention module is dot-multiplied with the output of the second activation layer to obtain the third visual-text feature.

[0075] Furthermore, the lightweight mask decoder of the embodiment of the present invention is used to generate a pixel-level segmentation mask according to the second visual-text feature and the third visual-text feature, and obtain a segmentation result according to the pixel-level segmentation mask. Using the existing lightweight mask decoder, the embodiment of the present invention comprehensively considers the output of the pixel-level attention module and the language gating module, that is, comprehensively considers the second visual-text feature and the third visual-text feature to generate a pixel-level segmentation mask, and a more accurate segmentation result can be obtained according to the pixel-level segmentation mask.

[0076] Furthermore, during the training process, the embodiment of the present invention constructs a total loss function composed of cross entropy loss and contrastive learning loss, and adds the contrastive learning loss as an additional loss term to the total loss. The total loss function formula is expressed as:

[0077] TotalLoss=SLoss+λ×CLoss;

[0078] Specifically, TotalLoss represents the total loss function, SLoss represents the main loss function for the remote sensing reference image segmentation task, such as the cross entropy loss, λ represents the weight of the contrastive learning loss. In practical applications, λ is usually set to a smaller value, such as 0.1 to 0.5, and CLoss represents the contrastive learning loss function. During the training process, the contrastive learning loss is optimized together with the main loss function of image segmentation.

[0079] Among them, the contrastive learning loss function is expressed as:

[0080]

[0081] Where N represents the number of training samples in the training sample set, x i represents the i-th training sample in the training sample set, x i + Represents the positive and negative sample pairs and the training sample x i The positive samples that make up the positive sample pair, f(x i ) represents x i After the output corresponding to the remote sensing reference image segmentation framework model, f(x i+ ) represents x i + After the output corresponding to the remote sensing reference image segmentation framework model, sim(f(xi),f(xi+)) means to find f(xi) and f(x i + ), K represents the number of negative samples in the positive and negative sample pairs, x j - represents the jth negative sample in the positive and negative sample set, f(x j - ) represents x j - After the output corresponding to the remote sensing reference image segmentation framework model, sim(f(xi),f(xj-)) represents the calculation of f(x) and f ( xj- ) The similarity between them is usually defined using cosine similarity, and τ represents the balance coefficient, which is used to adjust the smoothness of the distribution of positive and negative samples.

[0082] In the training process, the embodiment of the present invention incorporates contrastive learning loss as an additional loss term into the total loss function. By performing contrastive learning on the expanded data set, the calculated contrastive learning loss is jointly optimized with the image segmentation loss function. This method significantly enhances the model's ability to distinguish features and robustness, and improves the model's stability and accuracy when processing complex data.

[0083] S30, inputting the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain a segmentation result.

[0084] The embodiment of the present invention first obtains a remote sensing reference image to be segmented and its corresponding text information, wherein the remote sensing reference image to be segmented and its corresponding text information are from a RefSegRS dataset or an RRSIS-D dataset. Then, the remote sensing reference image to be segmented and its corresponding text information are input into a trained remote sensing reference image segmentation framework model to obtain a segmentation result, wherein the segmentation result includes a ground object category label of each pixel in the remote sensing reference image and a ground object category label probability value of each pixel.

[0085] In summary, the remote sensing reference image segmentation method based on contrastive learning and cross-attention proposed in the embodiment of the present invention is specially designed for the segmentation of reference images in the remote sensing field. The method combines the existing remote sensing reference image dataset and effectively expands the remote sensing reference image dataset by performing fine-grained Gaussian noise enhancement on the existing remote sensing reference image dataset, thereby improving the generalization ability of the model. In addition, the expanded remote sensing reference image dataset generates a set of positive and negative sample pairs, which are used for contrastive learning training, thereby calculating the contrastive learning loss and using it as part of the total training loss. This method enhances the model's ability to distinguish features and robustness. Furthermore, in the stage of fusion of visual features and text features, the embodiment of the present invention introduces a multi-head cross-attention module, a pixel-level attention module and language gating, which further improves the depth and accuracy of feature fusion. By using a multi-head cross-attention mechanism and a single-head scaling attention mechanism, the complex relationship between images and texts can be better captured, thereby improving the accuracy and reliability of image segmentation tasks. In general, the embodiments of the present invention comprehensively consider the supplementation of training sample sets, the design of loss functions, and the design of models during the training process. It not only exhibits better performance when processing remote sensing reference images, but also has stronger adaptability and higher effect stability, making it suitable for processing high-resolution visible light remote sensing images collected by satellites or aircraft, so as to achieve accurate segmentation of land object categories under language guidance.

[0086] Second, see Figure 4 , an embodiment of the present invention provides a remote sensing reference image segmentation device based on contrastive learning and cross attention, the device comprising:

[0087] A data generation module is used to obtain a training sample set, perform fine-grained Gaussian noise enhancement on the training sample set, and generate a positive and negative sample pair set based on the unenhanced training sample set and the enhanced training sample set; wherein each training sample in the training sample set includes a remote sensing reference image used for training and its corresponding text information;

[0088] A training module is used to input the training sample set and the positive and negative sample pair set into the initial remote sensing reference image segmentation framework model for training, so as to obtain a trained remote sensing reference image segmentation framework model; wherein, during the training process, a total loss function composed of a cross entropy loss and a contrastive learning loss is constructed, the training sample set is used to calculate the cross entropy loss, and the positive and negative sample pair set is used to calculate the contrastive learning loss;

[0089] The segmentation module is used to input the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain the segmentation result; wherein,

[0090] The remote sensing reference image segmentation framework model includes a visual backbone feature extraction network, a text feature extraction network, a multi-head cross-attention module, a pixel-level attention module, a language gating module, and a lightweight mask decoder; wherein the visual backbone feature extraction network is used to perform multi-scale feature extraction on the remote sensing reference image to be segmented to obtain visual extraction features; the text feature extraction network is used to perform feature extraction on the text information corresponding to the remote sensing reference image to be segmented to obtain text extraction features; the multi-head cross-attention module is used to fuse the visual extraction features and the text extraction features using the multi-head cross-attention mechanism to obtain the first visual-text feature; the pixel-level attention module is used to further fuse the first visual-text feature using the single-head scaled dot product attention mechanism to obtain the second visual-text feature; the language gating module is used to obtain the third visual-text feature by controlling the flow of text features in the second visual-text feature in the model; the lightweight mask decoder is used to generate a pixel-level segmentation mask according to the second visual-text feature and the third visual-text feature, and obtain the segmentation result according to the pixel-level segmentation mask.

[0091] As for the device embodiment of the second aspect, since it is basically similar to the method embodiment of the first aspect, the description is relatively simple, and the relevant parts may refer to the partial description of the method embodiment of the first aspect.

[0092] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0093] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the specification and its drawings. In the specification, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude multiple situations. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0094] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A remote sensing reference image segmentation method based on contrastive learning and cross attention, characterized in that: The method comprises: Acquire a training sample set, perform fine-grained Gaussian noise enhancement on the training sample set, and generate a positive and negative sample pair set according to the unenhanced training sample set and the enhanced training sample set; wherein each training sample in the training sample set includes a remote sensing reference image used for training and its corresponding text information; Inputting the training sample set and the positive and negative sample pair set into the initial remote sensing reference image segmentation framework model for training to obtain a trained remote sensing reference image segmentation framework model; wherein, during the training process, a total loss function consisting of a cross entropy loss and a contrastive learning loss is constructed, the training sample set is used for calculating the cross entropy loss, and the positive and negative sample pair set is used for calculating the contrastive learning loss; Inputting the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain the segmentation result; The remote sensing reference image segmentation framework model includes a visual backbone feature extraction network, a text feature extraction network, a multi-head cross-attention module, a pixel-level attention module, a language gating module, and a lightweight mask decoder; wherein the visual backbone feature extraction network is used to perform multi-scale feature extraction on the remote sensing reference image to be segmented to obtain visual extraction features; the text feature extraction network is used to perform feature extraction on the text information corresponding to the remote sensing reference image to be segmented to obtain text extraction features; the multi-head cross-attention module is used to fuse the visual extraction features and the text extraction features using a multi-head cross-attention mechanism to obtain a first visual-text feature; the pixel-level attention module is used to further fuse the first visual-text feature using a single-head scaled dot product attention mechanism to obtain a second visual-text feature; the language gating module is used to obtain a third visual-text feature by controlling the flow of text features in the second visual-text feature in the model; the lightweight mask decoder is used to generate a pixel-level segmentation mask according to the second visual-text feature and the third visual-text feature, and obtain a segmentation result according to the pixel-level segmentation mask.

2. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that: Performing fine-grained Gaussian noise enhancement on the training sample set includes: Fine-grained Gaussian noise enhancement is performed on the remote sensing reference images in the training sample set, and fine-grained Gaussian noise enhancement is not required for text information corresponding to the remote sensing reference images in the training sample set.

3. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that: The visual backbone feature extraction network is constructed by one or more of ResNet, MiT, PVTv2, LSKNet, Swin Transformer, Swin Transformer V2, ConvNeXt, UniRepLKNet, InternImage, FocalNet, VMamba, VisionTransformer series backbone networks and their subdivided backbones.

4. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that: The visual backbone feature extraction network includes 63 backbone networks, which are constructed by ResNet, MiT, PVTv2, LSKNet, Swin Transformer, Swin Transformer V2, ConvNeXt, UniRepLKNet, InternImage, FocalNet, VMamba, Vision Transformer series backbone networks and their subdivided backbones.

5. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that: The text feature extraction network is implemented by a Bert encoder.

6. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that: The multi-head cross attention module uses a multi-head cross attention mechanism to fuse the visual extraction feature and the text extraction feature to obtain a first visual-text feature, including: Using the visual extraction feature map as a query to form a first query matrix, and using the text extraction feature map as a key and a value to form a first key matrix and a first value matrix; Calculating an attention score of each attention head according to the visual extraction features and the text extraction features, the first query matrix and the first key matrix; Normalize the attention score of each attention head using the Softmax function, and calculate the weighted average of each attention head according to the first value matrix and the normalized attention score of each attention head; The weighted average of all attention heads is concatenated and linearly transformed to obtain the first visual-text feature.

7. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that: The pixel-level attention module further fuses the first visual-text features using a single-head scaled dot product attention mechanism to obtain a second visual-text feature, including: Aggregating the text features in the second visual-text features at each spatial position in a word-level dimension to form a sentence-level feature, and the sentence-level features at all spatial positions constitute a new text feature; Using the visual feature mapping in the second visual-textual feature as a query to form a second query matrix, and using the new text feature mapping in the second visual-textual feature as a key and a value to form a second key matrix and a second value matrix; Calculating a single-head attention score according to the visual features and the new text features in the second visual-text features, the second query matrix, and the second key matrix; Normalizing the single-head attention score by using a Softmax function, and calculating a weighted average value according to the second value matrix and the normalized single-head attention score; The weighted average is linearly transformed to obtain a second visual-text feature.

8. The remote sensing reference image segmentation method based on contrastive learning and cross attention according to claim 1, characterized in that: The remote sensing reference images to be segmented and their corresponding text information, as well as the remote sensing reference images used for training and their corresponding text information, are all from the RefSegRS dataset or the RRSIS-D dataset.

9. A remote sensing reference image segmentation device based on contrastive learning and cross attention, characterized in that: The device comprises: A data generation module, used to obtain a training sample set, perform fine-grained Gaussian noise enhancement on the training sample set, and generate a positive and negative sample pair set according to the unenhanced training sample set and the enhanced training sample set; wherein each training sample in the training sample set includes a remote sensing reference image used for training and its corresponding text information; A training module, used for inputting the training sample set and the positive and negative sample pair set into the initial remote sensing reference image segmentation framework model for training, so as to obtain a trained remote sensing reference image segmentation framework model; wherein, during the training process, a total loss function consisting of a cross entropy loss and a contrastive learning loss is constructed, the training sample set is used for calculating the cross entropy loss, and the positive and negative sample pair set is used for calculating the contrastive learning loss; A segmentation module is used to input the remote sensing reference image to be segmented and its corresponding text information into the trained remote sensing reference image segmentation framework model to obtain the segmentation result; The remote sensing reference image segmentation framework model includes a visual backbone feature extraction network, a text feature extraction network, a multi-head cross-attention module, a pixel-level attention module, a language gating module, and a lightweight mask decoder; wherein the visual backbone feature extraction network is used to perform multi-scale feature extraction on the remote sensing reference image to be segmented to obtain visual extraction features; the text feature extraction network is used to perform feature extraction on the text information corresponding to the remote sensing reference image to be segmented to obtain text extraction features; the multi-head cross-attention module is used to fuse the visual extraction features and the text extraction features using a multi-head cross-attention mechanism to obtain a first visual-text feature; the pixel-level attention module is used to further fuse the first visual-text feature using a single-head scaled dot product attention mechanism to obtain a second visual-text feature; the language gating module is used to obtain a third visual-text feature by controlling the flow of text features in the second visual-text feature in the model; the lightweight mask decoder is used to generate a pixel-level segmentation mask according to the second visual-text feature and the third visual-text feature, and obtain a segmentation result according to the pixel-level segmentation mask.

Images