A semantic segmentation method for ultra-high resolution remote sensing images based on balanced anchor point sampling and prototype representation

Abstract

This invention discloses a semantic segmentation method for ultra-high resolution remote sensing images based on balanced anchor sampling and prototype representation. It combines text features provided by a pre-trained visual language model with traditional segmentation models, including steps such as data preprocessing, multi-size anchor region cropping, target detection and bounding box sorting, visual feature extraction, text feature extraction, prototype representation injection, and model training. This invention improves the overall performance of the model by pre-extracting complete ground features and injects text information into the traditional semantic segmentation model to perform detailed semantic segmentation of ultra-high resolution remote sensing images, thereby enhancing the performance and robustness of the ultra-high resolution remote sensing image semantic segmentation model.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image semantic segmentation technology, specifically relating to a method for semantic segmentation of ultra-high resolution remote sensing images based on balanced anchor point sampling and prototype representation. Background Technology

[0002] Ultra-high resolution (UHR) imagery has developed rapidly in recent years, mainly due to advancements in remote sensing technology and improved satellite image acquisition capabilities. These images are becoming increasingly important in multiple fields because of their fine details and rich ground feature information.

[0003] In recent years, visual language pre-trained models have achieved remarkable results in the field of remote sensing, such as GeoRSCLIP [Zhang Z, Zhao T, Guo Y, et al. RS5M and GeoRSCLIP: A large scale vision-language dataset and a large vision-language model for remote sensing[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024], RemoteCLIP [Liu F, Chen D, Guan Z, et al. Remoteteclip: A vision language foundation model for remote sensing[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024], and SAM [Kirillov A, Mintun E, Ravi N, et al. Segment anything[C] / / Proceedings of the IEEE / CVF International Conference on Computer Science and Technology]. [Vision.2023:4015-4026] etc., these models have achieved remarkable results on specific tasks and image or image-text datasets; however, when the image data is extended to the land cover segmentation of ultra-high resolution images, the performance of the models drops sharply due to the limitations of the pre-training task and the change in image resolution, and the results are unsatisfactory. The main challenges come from two aspects: First, existing remote sensing image land cover segmentation models can only process data by random cropping during the training phase, or cut the entire ultra-high resolution image into fixed-size blocks for training, which usually cannot capture complete land cover information, nor can it apply high-quality text information for feature enhancement; Second, existing pre-trained visual language models in the remote sensing field are usually pre-trained on multi-source data, which is not effective in the transfer of downstream tasks, and the training cost is usually high.

[0004] In a typical ultra-high resolution remote sensing image land cover segmentation task, the main challenge is the enormous size of the images. Limited by existing hardware and algorithms, end-to-end training and inference on a single image is impossible. A common solution is to segment the ultra-high resolution image into smaller, fixed-size images, transform the dataset, and feed it into the semantic segmentation model for training. During inference, a sliding window algorithm is used to progressively predict the entire image to obtain the final segmentation result. This training method leads to two serious problems: First, it maintains the original data category distribution throughout the training process. If the dataset has a long tail problem, the head categories will overfit during training, reducing generalization ability, while the tail categories will remain underfitted. Second, the model struggles to see complete land covers, always relying on texture features to distinguish them, lacking shape and other feature information. This makes it difficult to distinguish subcategories such as lakes and rivers, residential areas and industrial areas.

[0005] Existing methods for segmentation of ultra-high resolution remote sensing images include a multi-branch network that extracts features at different scales. One network extracts local features, while another extracts contextual features. Then, a feature fusion module and a feature refinement module are designed to aggregate the contextual features into the local features, resulting in an enhanced feature. This enhanced feature is then used for subsequent processing. Examples include GLNet [Zhang S, Song L, Gao C, et al. GLNet: Global local network for weakly supervised action localization [J]. IEEE Transactions on Multimedia, 2019, 22(10): 2610-2622], FctlNet [Li Q, Yang W, Liu W, et al. From contexts to locality: Ultra-high resolution image segmentation via locality-aware contextual correlation [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 7252-7261], and WiCoNet [Ding L, Lin D, Lin S, et al. Looking outside the window:Wide-context transformer for the semanticsegmentation of high-resolution remote sensing images.arXiv 2021[J].arXivpreprint arXiv:2106.15754]、ISDNet[Guo S,Liu L,Gan Z,et al.Isdnet:Integratingshallow and deep networks for efficient ultra-high resolution segmentation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2022:4361-4370] and other methods all adopt this idea.Another approach is to reduce the computational complexity of the network by processing an entire remote sensing satellite image in a single forward pass. This method typically employs traditional feature extraction methods such as wavelet transform, combined with lightweight convolutional neural networks, such as WSDNet [Ji D, Zhao F, Lu H, et al. Ultra-high resolution segmentation with ultra-rich context: A novelbenchmark[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2023:23621-23630] and GPWFormer [Ji D, Zhao F, Lu H. Guided patch-grouping wavelet transformer with spatial congruence for ultra-high resolution segmentation[J]. arXiv preprint arXiv:2307.00711,2023]. Based on these two methods, methods such as patch selection and scale-adaptive selection have been derived and have made some progress. However, a patch is often a small pixel block of 14×14 or 32×32. The method of this invention is based on a larger anchor point region and can provide richer contextual information.

[0006] Our research revealed that in the field of ultra-high resolution image segmentation, existing segmentation models (such as Deeplabv3+ [Chen LC, Zhu Y, Papandreou G, et al. Encoder-decoder with atrousseparable convolution for semantic image segmentation [C] / / Proceedings of the European conference on computer vision (ECCV). 2018:801-818], Segformer [Xie E, Wang W, Yu Z, et al. SegFormer: Simple and efficient design for semanticsegmentation with transformers [J]. Advances in neural information processing systems, 2021, 34:12077-12090], etc.) have not yet been combined with the multimodal features provided by pre-trained text image models (such as GeoRSCLIP, RemoteCLIP, etc.). The former usually provides high-quality segmentation results within a specific domain, but lacks rich knowledge of the entire domain; the latter usually contains rich knowledge and is good at capturing the category features of objects, but it is difficult to generate high-quality pixel-level classification results. Since the rich information in ultra-high resolution remote sensing images is difficult to feed into the training of traditional segmentation models, we need to inject the information of pre-trained visual language models as prototype representations into traditional segmentation models to improve the performance of semantic segmentation models. Summary of the Invention

[0007] In view of the above, this invention provides a semantic segmentation method for ultra-high resolution remote sensing images based on balanced anchor point sampling and prototype representation. It proposes a novel multi-size anchor point region cropping method, which can better capture multi-scale contextual features and samples according to the principle of prioritizing a few categories and prioritizing high category richness. At the same time, it can combine a pre-trained visual language model with a traditional segmentation model to perform detailed semantic segmentation of ultra-high resolution remote sensing images, thereby improving the performance and robustness of the semantic segmentation model for ultra-high resolution remote sensing images.

[0008] A semantic segmentation method for ultra-high resolution remote sensing images based on balanced anchor point sampling and prototype representation includes the following steps:

[0009] (1) Preprocessing and data augmentation of the original ultra-high resolution image are performed to obtain the enhanced image I. aug ;

[0010] (2) For image I aug Multi-scale clipping is performed based on the anchor point region to obtain a multi-size anchor point region clipping group;

[0011] (3) Using a pre-trained target detection model to analyze image I aug The detection process generates multiple remote sensing feature frames.

[0012] (4) Sort the remote sensing bounding boxes according to the rarity of the main category and the richness of the category. Based on the sorting results, mix the multi-size anchor point region cropping group with the image of the remote sensing bounding box as a batch of training samples.

[0013] (5) Obtain a large number of remote sensing-specific land cover vocabulary descriptions, and obtain a remote sensing text vocabulary library T by classifying and deduplicating the vocabulary t in the library. Fill the vocabulary t in the library into a pre-prepared template to obtain the corresponding text description T. t This will then describe the text T. t Input is fed into a pre-trained visual language model to extract the corresponding prototype representation F. t , t∈T;

[0014] (6) Constructing a semantic segmentation model for remote sensing images, which includes:

[0015] A visual encoder is used to extract visual features from an image;

[0016] The semantic segmentation decoder performs semantic segmentation based on the features fused from visual and textual data to obtain the image classification results;

[0017] (7) The above model is trained by combining training samples with prototype representation. Finally, the ultra-high resolution image to be classified is processed and input into the trained model to predict the output and obtain the corresponding classification result.

[0018] Furthermore, the specific implementation of step (1) is as follows: First, the original ultra-high resolution image is preprocessed, retaining only the RGB three-channel bands to obtain the processed image I. rgb Then for image I rgb Data augmentation processing includes random flipping, rotation, color space dithering, image sharpening, and random noise transformation to obtain the enhanced image I. aug .

[0019] Furthermore, the specific implementation of step (2) is as follows:

[0020] 2.1 In image I augRandomly select a point in the middle, and use this point as the top left vertex to cut a fixed-size area, namely the anchor point area. The anchor point area must have at least 2 categories, and the pixel category with the highest proportion cannot be the background and its proportion cannot exceed h (0 < h < 1) times the total area of ​​the anchor point area.

[0021] 2.2 Define a series of cutting scales [l2, l3, ..., l K For any scale l k At this scale, the cropped image size is k times the size of the anchor point region, where k is a natural number and 2≤k≤K, and K is a natural number greater than 2.

[0022] 2.3 According to scale l k The image size of the cropped image is in image I aug The search is performed with a horizontal step size of (k-1)crop_w and a vertical step size of (k-1)crop_h. The searched image must completely contain the anchor point region, where crop_h and crop_w are the height and width of the anchor point region, respectively. Then, one image is randomly selected from the searched images as the scale l. k Cropped Image I lk The search process involves traversing the image according to the step size mentioned above, recording the image blocks containing the anchor point region, and obtaining the searched images. The specific number of images searched varies with the size of the original image.

[0023] 2.4 Following steps 2.1 to 2.3, a multi-size anchor point region cropping group is obtained, which includes the anchor point region image I. l1 And cropped images at multiple scales [I l2 ,I l3 ,…,I lk ].

[0024] Furthermore, in step (3), an instance segmentation model (such as SAM-HQ [Ke L, Ye M, Danelljan M, et al. Segment anything in high quality [J]. Advances in Neural Information Processing Systems, 2024, 36]) is used to detect image I. aug Instances in the data are selected, and a threshold is set to filter out instances with high confidence. A series of remote sensing bounding boxes are generated based on the instance mask, and each remote sensing bounding box is represented by a quadruple (w,h,c). x ,c y It is represented in the form of ), where w and h are the width and height of the remote sensing feature frame, respectively, and c x ,c y These are the coordinates of the center point of the remote sensing ground feature frame.

[0025] Furthermore, the specific implementation of step (4) is as follows:

[0026] 4.1 Based on the label statistics, the pixel proportion of each category in the original ultra-high resolution image is calculated. Each remote sensing object frame has one main category and several subcategories. The main category is the category with the highest pixel proportion within the remote sensing object frame.

[0027] 4.2 Sort all remote sensing bounding boxes according to the rarity of the main category and the richness of the category. Prioritize the comparison of the rarity of the main category, that is, the remote sensing bounding box with the smaller pixel proportion of the main category in the original ultra-high resolution image is ranked higher. When the rankings are the same, compare the richness of the category, that is, the remote sensing bounding box with more categories is ranked higher.

[0028] 4.3 Determine the number of images to be mixed between the multi-size anchor point region cropping group and the remote sensing feature frame according to a certain ratio. Select the corresponding number of remote sensing feature frames according to the sorting results. Mix the images of these remote sensing feature frames with the images in the multi-size anchor point region cropping group as a batch of training samples and unify the image size by resampling. The remote sensing feature frames that are ranked higher have a higher probability of being sampled.

[0029] Furthermore, in step (5), based on the category of the word t, t is filled into the template to obtain the corresponding text description T. t The template is represented as a satellite image of {}, and then T t After being segmented by a word segmenter (such as CLIP), the text is converted from a string to an integer encoding and a series of special characters and padding masks are added to obtain the text prototype. Finally, the text prototype is input into a pre-trained visual language model (such as GeoRSCLIP) to extract the corresponding prototype representation F. t .

[0030] Further, in step (6), the visual encoder uses a backbone network (such as Segformer-mit-b5 [Xie E, Wang W, Yu Z, et al. SegFormer: Simple and efficient design for semanticsegmentation with transformers [J]. Advances in neural information processing systems, 2021, 34: 12077-12090]) to extract features from the image information. The input of the backbone network is an image sequence that is a mixture of multi-size anchor point region cropping groups and remote sensing object frames. After feature extraction and alignment of channel dimensions by the mapping layer, a multi-scale feature map, i.e., the visual features of the image, is obtained. Then, the multi-scale feature map is combined with the prototype representation F t Pairing and fusing are performed to obtain feature F it Then F it After being concatenated with the multi-scale feature map, it is input into the semantic segmentation decoder for semantic segmentation.

[0031] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described semantic segmentation method for ultra-high resolution remote sensing images based on balanced anchor sampling and prototype representation.

[0032] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described semantic segmentation method for ultra-high resolution remote sensing images based on balanced anchor sampling and prototype representation.

[0033] This invention provides a novel training method for ultra-high resolution image segmentation. In the process of training the semantic segmentation model, this invention uses multi-size anchor point region clipping to provide a multi-size field of view of a region. Furthermore, in order to balance the land cover categories, we use land cover detection to provide land cover regions that can be sampled for category as part of the input for model training.

[0034] In addition, to inject prototype information into the semantic segmentation model, this invention proposes a novel information injection strategy, including remote sensing-specific land cover vocabulary integration, prototype representation extraction from a remote sensing-specific pre-trained visual language model, and a feature fusion module. During the training process of the semantic segmentation model, this invention maintains the model training loss as cross-entropy loss and selects the model with the best training score in multiple rounds of training. Experiments have verified that the model with the best score performs best on ultra-high resolution images. Therefore, based on a remote sensing-specific pre-trained visual language model, this invention achieves higher performance through text information injection and reselection of training images, greatly improving the accuracy and robustness of remote sensing ultra-high resolution image segmentation. Attached Figure Description

[0035] Figure 1 This is a schematic diagram illustrating the steps of the ultra-high resolution remote sensing image semantic segmentation method of the present invention.

[0036] Figure 2 This is a schematic diagram of the specific system implementation process in the embodiment of the method of the present invention. Detailed Implementation

[0037] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] like Figure 1 and Figure 2 As shown, the present invention provides a semantic segmentation method for ultra-high resolution remote sensing images based on balanced anchor point sampling and prototype representation, comprising the following steps:

[0039] (1) Preprocess the original ultra-high resolution image I, retaining only the RGB three channels, to obtain the processed image I. rgb .

[0040] Remote sensing satellite images may typically contain multiple bands. For example, Gaofen-2 satellite images contain not only the RGB band of traditional images but also the near-infrared band; Sentinel-2 satellite images contain not only the RGB band but also near-infrared, short-infrared, and other bands.

[0041] Since current mainstream computer vision networks process images in RGB format, and adding non-RGB bands to existing models severely impacts their generalization ability, this implementation method requires band selection during the data processing stage to convert the image to RGB format. rgb .

[0042] (2) For image I rgbData augmentation processing can be performed, and available data augmentation strategies include random flipping, rotation, color space dithering, random cropping, image sharpening, and random noise transformation to obtain the enhanced image I. aug .

[0043] To improve the generalization ability of the model and increase the diversity of input data, we usually need to perform some data augmentation operations on the images. For remote sensing images, random flipping and rotating operations are important because remote sensing features often require the model to learn rotation-invariant properties. Common data preprocessing operations also include image sharpening and random noise transformation. In this implementation, to verify the generality of the method, only random flipping and rotating operations were used. The specific operation results are as follows:

[0044] I aug =rotate(random_flip(I rgb ))

[0045] The expression for random_flip is as follows, where c represents the color channel of the image. First, a random number p between 0 and 1 is generated, corresponding to the random operation.

[0046]

[0047] The expression for rotate is as follows, where c represents the color channel of the image:

[0048]

[0049] Where (x0, y0) are the coordinates of the origin of rotation, and θ is the rotation angle.

[0050] (3) Using the designed multi-size anchor point region clipping method, with the crop_size scale as the reference, for I aug Perform multi-scale cropping to obtain a multi-size anchor point region cropping group containing images at k scales. Perform the same operation on the labels.

[0051] This invention performs a series of optimizations for random cropping to adapt to ultra-high resolution images, namely, multi-size anchor point region cropping, specifically:

[0052] Because annotation of ultra-high resolution remote sensing images is extremely costly, expert annotators typically only select a subset of highly relevant features for pixel-by-pixel annotation, while labeling a large number of uninterested areas as background. Therefore, after determining the crop region size (crop_h, crop_w), a carefully selected anchor region is necessary. We select our initial anchor region by limiting it to at least two categories, ensuring the most prevalent category is not background and its proportion does not exceed 75% of the crop region. Since this region maintains the ground sampling rate of the image, it is particularly important for the model to capture details; we call it I0. l1 Other scales of cropping are all done through I l1 To obtain.

[0053] This implementation specifies a series of cropping scales for cropping the image, I lk Indicates compared to I l1 The cropped image is k times larger than the original image size. The cropping method for each scale is as follows:

[0054]

[0055] Where: crop_h, crop_w are I l1 The dimensions of multi_crop_h and multi_crop_w are I lk In this embodiment, the size of crop_h = crop_w = 512.

[0056] To meet the above conditions, an appropriate trimming step size needs to be selected for each scale. The trimming step size should satisfy the following:

[0057] stride_h∈[crop_h,(k-1)*crop_h]; stride_w∈[crop_w,(k-1)*crop_w]

[0058] In the specific implementation process, we take stride_h = (k-1)*crop_h, stride_w = (k-1)*crop_w, and take 4 levels, k∈[1,2,3,4], for I aug Perform cropping to obtain cropped multi-size images [I] l1 ,I l2 ,I l3 ,I l4 ].

[0059] (4) Use the target detection model to obtain the complete remote sensing object frame.

[0060] In recent years, with the rapid development of computer vision in the field of remote sensing, many remote sensing ground object detection models have emerged, such as Rotated Faster R-CNN [Yang S, Pei Z, Zhou F, et al. Rotated faster R-CNN for oriented object detection in aerial images[C] / / Proceedings of the 2020 3rd International Conference on Robot Systems and Applications.2020:35-39], RVSA [Wang D, Zhang Q, Xu Y, et al. Advancing plain vision transformer toward remote sensing foundation model[J].IEEE Transactions on Geoscience and RemoteSensing,2022,61:1-15], and Oriented R-CNN [Xie X, Cheng G, Wang J, et al. Oriented R-CNN for object detection[C] / / Proceedings of the IEEE / CVF international conferenceon computer While models like those in Vision. 2021:3520-3529 exist, most are designed for aerial images taken at low altitudes or satellite images with few ground features, making them unsuitable for detecting complete ground features. The target detection model used in this invention should be able to detect some complete ground features in the original ultra-high resolution remote sensing image to a certain extent, ensuring the integrity of the training area. With the emergence of SAM models in the instance segmentation domain, obtaining boxes using masks has become possible, and SAM series models are naturally more suitable for processing high-resolution images. This implementation uses the SAM-HQ model to generate multiple detection boxes, each box consisting of a quadruple (w, h, c). x ,c y This can be represented by ), specifically mathematically as:

[0061] (w,h,c x ,c y )=f θ (I aug )

[0062] Where: f θ () is the mapping function of the SAM-HQ neural network, where w, h, and c are in the quadruple.x ,c y These represent the width, height, and center point coordinates of the detection box, respectively.

[0063] (5) Sort the detected complete remote sensing feature frames.

[0064] This invention requires prior knowledge of the category distribution of the entire dataset to obtain the proportion of pixels in each category. The smaller the proportion, the higher the category rarity. Since object detection results are usually complete instances, each detection box first has a primary category and several secondary categories. This implementation sets two attributes for each detection box: primary category rarity and category richness (i.e., the number of category types). All detection boxes are sorted according to the primary category rarity, with detection boxes of rarer primary categories ranking higher. When the primary category rarity is the same, detection boxes with higher category richness rank higher, thus achieving balanced category sampling. During training, detection boxes with higher rankings are more likely to be used as training data. Finally, we obtain a series of images cropped from the detection boxes [I]. box1 ,I box2 ,…,I boxn ].

[0065] (6) In this embodiment, we take a set of cropped images of multiple anchor point regions [I], which are mixed according to a certain ratio of cropped images and remote sensing feature frames. l1 ,I l2 ,I l3 ,I l4 Then select four detection boxes [I] box1 ,I box2 ,I box3 ,I box4 During the selection process, samples with high class rarity and high class richness are prioritized to ensure that the top-ranked detection boxes are more likely to be selected, thus obtaining a batch of training samples:

[0066] I train =[I l1 ,I l2 ,I l3 ,I l4 ,I box1 ,I box2 ,I box3 ,I box4 ]

[0067] Regarding the scale of the detection box, if the size of the detection box itself is larger than (crop_h,crop_w), the image is cropped from the top left corner to a scale of (crop_h,crop_w) as training data; if the size of the detection box itself is smaller than (crop_h,crop_w), the detection box size is resampled to (crop_h,crop_w) and used as training data.

[0068] (7) Transfer the training data I train The data is fed into a backbone-based visual encoder for feature extraction, obtaining features from k stages [F]. o1 ,F o2 ,…F ok The above features are mapped and aligned in dimensions through k mapping layers to obtain multi-scale feature maps [F1, F2, ... F]. k ], used for the final land feature classification.

[0069] Commonly used backbone networks include Resnet[He K, Zhang in neural information processingsystems,2021,34:12077-12090], Mobilenet[Howard AG. Mobilenets: Efficientconvolutional neural networks for mobile vision applications[J].arXivpreprint arXiv:1704.04861,2017], SwinTransformer[Liu Z,Lin Y,Cao Y,et al.Swintransformer:Hierarchical vision transformer using shifted windows[C] / / Proceedings of the [IEEE / CVF international conference on computer vision. 2021:10012-10022] etc. In this embodiment, the backbone network is implemented using Segformer-mit-b5, but it is not limited to methods based on Segformer-mit. The Segformer-mit backbone network is a semantic segmentation network backbone architecture based on Transformer, which accepts RGB three-channel image input and outputs four feature maps at different scales. The downsampling scale compared to the original image is [missing information]. The Segformer-mit architecture uses EfficientSelf-Attn and Mix-FFN modules to extract features, which can reduce computational complexity. In each stage, Overlap Patch Merging is used to aggregate and downsample features. After obtaining feature maps for multiple stages, the model aligns the number of channels through four 1×1 convolutional modules.

[0070] This implementation uses an input image of size (512, 512) to obtain multi-scale feature maps [F1, F2, ... F1] with feature sizes of [128, 64, 32, 16]. k ].

[0071] (8) Obtain a series of remote sensing-specific land cover vocabulary descriptions, map these vocabulary words to the categories of each dataset to enrich the category information, and then integrate them to obtain the remote sensing text vocabulary library T.

[0072] This implementation provides detailed text descriptions and classifications for each subject category, and obtains a number of subcategories as shown in Table 1, which are provided as text prompts to a pre-trained visual language model specifically for the remote sensing field.

[0073] Table 1

[0074]

[0075]

[0076] (9) Use text information and pre-trained visual language models to obtain text features.

[0077] The remote sensing pre-trained visual language model selected in this embodiment is GeoRSCLIP. GeoRSCLIP uses OpenCLIP (https: / / github.com / mlfoundations / open_clip.git) weights to initialize the weights of the CLIP model. It is trained on the large-scale remote sensing image-text pair dataset RS5M [Zhang Z, Zhao T, Guo Y, et al. RS5M and GeoRSCLIP: A large scale vision-language dataset and a large vision-language model for remote sensing[J].IEEE Transactions on Geoscience and RemoteSensing,2024], and can extract text features with a large amount of remote sensing knowledge. After obtaining the text vocabulary T in the previous step, each word description in the vocabulary is input into the CLIP [Radford A, Kim JW, Hallacy C, et al. Learning transferable visual models from natural language supervision[C] / / International conference on machine learning.PMLR,2021:8748-8763] standard template "asatellite image of {}" to obtain multiple text prototypes. The above N templates are then processed by the CLIP tokenizer to obtain tokens. The text is converted from a string to an integer encoding and a series of special characters and padding masks are added to obtain the text token F_token. This token is then sent to the text branch of GeoRSCLIP to obtain the prototype representation F. t .

[0078] (10) Combine the multi-scale image features [F1, F2, F3, F4] with the prototype representation F t The image and text feature fusion module is used to obtain the fused feature F. it , will F it The multi-scale image features [F1, F2, F3, F4] are concatenated and fed into the semantic segmentation decoder to obtain the classification result.

[0079] Since we do not have text labels for each individual image, this implementation keeps the text features unchanged, aligns the image features and text features in the same dimension, and calculates the cosine similarity to obtain the pairing probability of each image feature and each text feature.

[0080] In this embodiment, the features [F1, F2, F3, F4] are first resampled and aligned to their dimensions, then concatenated to obtain the multi-scale fused feature F. if After passing through a layer of MLP and F t Align the dimensions, then standardize the features to limit the range of the feature vectors to between 0 and 1, and finally calculate the cosine similarity of each vector to obtain the final fused feature F with shape [b,n,h,w]. it Where b is the batch size for training, n is the number of texts, and h and w represent the height and width of the feature map, respectively; finally, F if and F it The final feature F is obtained by concatenating along dimension one and input into the semantic segmentation decoder. The entire feature fusion step is formally represented as follows:

[0081] F if =concate([F1,resized(F2),resized(F3),resized(F4)],dim=0)

[0082] F it =BMM(F t ,MLP(F if ) T )

[0083] F = concate([F if ,F it ],dim=1)

[0084] Where: resized represents bilinear interpolation, concatenates represent concatenation along a specified dimension, MLP is a two-layer fully connected network, BMM can be expressed by the following formula, and all three operations can be implemented by calling the torch library.

[0085] X′=MLP(X=(X T W1+b1) T W2+b2

[0086] P n×h×w =BMM(M c×h×w N c×n )

[0087]

[0088] (11) The model is trained using cross-entropy loss to obtain the final semantic segmentation model.

[0089] The training data used in this implementation are the publicly available dataset GID of ultra-high resolution remote sensing images from the GF-2 satellite and the dataset URUR from multi-source satellite images [Ji D, Zhao F, Lu H, et al. Ultra-highresolution segmentation with ultra-rich context: A novel benchmark [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023:23621-23630], where GID [Tong XY, Xia GS, Lu Q, et al. Land-cover classification with high-resolution remote sensing images using transferable deep models [J]. Remote Sensing of The dataset [Environment, 2020, 237:111322] has an image size of (6800, 7200), with 120 images in the training set and 30 images in the validation set; the URUR dataset has an image size of (5120, 5120), with 2157, 280, and 571 images in the training, validation, and test sets, respectively. The model was stopped after 160,000 iters, and a model checkpoint (complete model parameters) was saved every 8,000 iters (one forward and one backward propagation process during training). The AdamW optimizer was used for training, with a learning rate of 1e-6, betas of [0.9, 0.999], and weight_decay of 0.01. The learning rate decay strategy used linear decay (LinearLR) for the first 1500 iters and polynomial decay (PolyLR) for subsequent training.

[0090] The loss function used is K-type cross-entropy loss, as shown in the following formula:

[0091]

[0092] Where: p ic K represents the probability that the i-th sample is classified into the c-th category, where K refers to the number of categories and N refers to the total number of samples.

[0093] Table 2 shows the performance gains of this implementation compared to the basic Segformer-b5 model on the URUR and GID datasets. In the table, mc represents multi-level crop (mlrc), samhq represents the bounding boxes obtained by object detection using the SAM-HQ model, which are all resampled to a fixed scale before being fed into the model for training; keepgsd refers to the bounding boxes obtained by object detection using the SAM-HQ model, where bounding boxes that do not conform to the fixed size are cropped from the top left corner to a fixed size for training; clip-text indicates whether to add text features extracted by GeoRSCLIP, × indicates that this method is not used, and √ indicates that this method is used on the Segformer-b5 architecture.

[0094] Table 2

[0095]

[0096] The experimental results above show that this implementation method achieves a significant performance improvement compared to the basic Segformer-b5 model. Furthermore, cropping the detection boxes after applying the SAMHQ model to maintain the ground sampling distance further enhances model performance. This may be because, as an ultra-high resolution segmentation method, any form of resampling leads to pixel loss in the input image, resulting in suboptimal segmentation results. Adding text information extracted by GeoRSCLIP significantly improves model performance, demonstrating that pre-trained visual language models can indeed inject difficult-to-learn knowledge into semantic segmentation models, thereby promoting the model's learning of richer features.

[0097] Table 3 shows the IoU accuracy of this embodiment on a single category, further demonstrating the effectiveness of the invention, especially the significant role of text features in improving the segmentation accuracy of fewer categories in the dataset.

[0098] Table 3

[0099] Data unlabel building framland forest meadow water road greenhouse Bareland URUR 1.06 72.4 75.91 47.77 - 51.51 49.18 45.19 33.15 GID - 78.91 80.87 76.49 54.7 90.62 - - -

[0100] Table 4 shows a performance comparison between this embodiment and other methods in the field. As can be seen from the experimental results, the method of the present invention outperforms existing methods on all three datasets, demonstrating the superiority of the method of the present invention.

[0101] Table 4

[0102]

[0103] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A semantic segmentation method for ultra-high resolution remote sensing images based on balanced anchor point sampling and prototype representation, comprising the following steps: (1) Preprocessing and data augmentation of the original ultra-high resolution image are performed to obtain the enhanced image I. aug ; (2) For image I aug Multi-scale clipping is performed based on the anchor point region to obtain a multi-size anchor point region clipping group. The specific implementation method is as follows: 2.1 In image I aug Randomly select a point in the middle, and use this point as the top left vertex to crop a fixed-size area, namely the anchor point area. The anchor point area must have at least 2 categories, and the pixel category with the highest proportion cannot be the background and its proportion cannot exceed 0.75 times the total area of ​​the anchor point area. 2.2 Define a series of cutting scales [l2, l3, ..., l k For any scale l k At this scale, the cropped image size is k times the size of the anchor point region, where k is a natural number and 2≤k≤K, and K is a natural number greater than 2. 2.3 According to scale l k The image size of the cropped image is in image I aug The search is performed with a horizontal step size of (k-1)crop_w and a vertical step size of (k-1)crop_h. The searched image must completely contain the anchor point region, where crop_h and crop_w are the height and width of the anchor point region, respectively. Then, one image is randomly selected from the searched images as the scale l. k Cropped Image I lk ; 2.4 Following steps 2.1 to 2.3, a multi-size anchor point region cropping group is obtained, which includes the anchor point region image I. l1 And cropped images at multiple scales [I l2 ,I l3 ,…,I lK ]; (3) Using a pre-trained target detection model to analyze image I aug The detection process generates multiple remote sensing feature frames. (4) Sort the remote sensing bounding boxes according to the rarity and richness of the main category. Based on the sorting results, mix the multi-size anchor point region cropping group with the images of the remote sensing bounding boxes as a batch of training samples. The specific implementation method is as follows: 4.1 Based on the label statistics, the pixel proportion of each category in the original ultra-high resolution image is calculated. Each remote sensing object frame has one main category and several subcategories. The main category is the category with the highest pixel proportion within the remote sensing object frame. 4.2 Sort all remote sensing bounding boxes according to the rarity of the main category and the richness of the category. Prioritize the comparison of the rarity of the main category, that is, the remote sensing bounding box with the smaller pixel proportion of the main category in the original ultra-high resolution image is ranked higher. When the rankings are the same, compare the richness of the category, that is, the remote sensing bounding box with more categories is ranked higher. 4.3 Determine the number of images to be mixed between the multi-size anchor point region cropping group and the remote sensing feature frame according to a certain ratio. Select the corresponding number of remote sensing feature frames according to the sorting results. Mix the images of these remote sensing feature frames with the images in the multi-size anchor point region cropping group as a batch of training samples and unify the image size by resampling. The remote sensing feature frames that are ranked higher have a higher probability of being sampled. (5) Obtain a large number of remote sensing-specific land cover vocabulary descriptions, and obtain a remote sensing text vocabulary library T by classifying and deduplicating the vocabulary t in the library. Fill the vocabulary t in the library into a pre-prepared template to obtain the corresponding text description T. t This will then describe the text T. t Input is fed into a pre-trained visual language model to extract the corresponding prototype representation F. t , t∈T; (6) Constructing a semantic segmentation model for remote sensing images, which includes: A visual encoder is used to extract visual features from an image; The semantic segmentation decoder performs semantic segmentation based on the features fused from visual and textual data to obtain the image classification results; (7) The above remote sensing image semantic segmentation model is trained by combining training samples with prototype representation. Finally, the ultra-high resolution image to be classified is processed and input into the trained model to predict the output and obtain the corresponding classification result.

2. The ultra-high resolution remote sensing image semantic segmentation method according to claim 1, characterized in that: The specific implementation of step (1) is as follows: First, the original ultra-high resolution image is preprocessed, retaining only the RGB three-channel bands, to obtain the processed image I. rgb Then for image I rgb Data augmentation processing, including random flipping, rotation, color space dithering, image sharpening, and random noise transformation, is performed to obtain the enhanced image I. aug .

3. The ultra-high resolution remote sensing image semantic segmentation method according to claim 1, characterized in that: In step (3), an instance segmentation model is used to detect image I. aug Instances in the data are selected, and a threshold is set to filter out instances with high confidence. A series of remote sensing bounding boxes are generated based on the instance mask, and each remote sensing bounding box is represented by a quadruple (w,h,c). x ,c y It is represented in the form of ), where w and h are the width and height of the remote sensing feature frame, respectively, and c x ,c y These are the coordinates of the center point of the remote sensing ground feature frame.

4. The ultra-high resolution remote sensing image semantic segmentation method according to claim 1, characterized in that: In step (5), based on the category of the word t, t is filled into the template to obtain the corresponding text description T. t The template is represented as an asatellite image of {}, and then T t After being segmented by a word segmenter, the text is converted from a string to an integer encoding and a series of special characters and padding masks are added to obtain the text prototype. Finally, the text prototype is input into a pre-trained visual language model to extract the corresponding prototype representation F. t .

5. The ultra-high resolution remote sensing image semantic segmentation method according to claim 1, characterized in that: In step (6), the visual encoder uses a backbone network to extract features from the image information. The input to the backbone network is an image sequence that combines multi-size anchor point region cropping groups with remote sensing object frames. After feature extraction and alignment of channel dimensions by the mapping layer, a multi-scale feature map, i.e., the visual features of the image, is obtained. Then, the multi-scale feature map is compared with the prototype representation F. t Pairing and fusing are performed to obtain feature F it Then F it After being concatenated with the multi-scale feature map, it is input into the semantic segmentation decoder for semantic segmentation.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: The processor is used to execute the computer program to implement the ultra-high resolution remote sensing image semantic segmentation method as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the ultra-high resolution remote sensing image semantic segmentation method as described in any one of claims 1 to 5.

Images