Vision transformer-based remote sensing image segmentation method and device, medium and equipment

Abstract

The application discloses a remote sensing image segmentation method and device based on a visual Transformer, a medium and equipment. The method comprises the following steps: acquiring a remote sensing image dataset, wherein the remote sensing image dataset comprises a query remote sensing image and a support remote sensing image; inputting the remote sensing dataset into a remote sensing image segmentation model to obtain a predicted remote sensing image segmentation result; inputting the query remote sensing image and the support remote sensing image into an adjustable branch to obtain an adjustable branch prototype vector; inputting the query remote sensing image and the support remote sensing image into a frozen branch to obtain a frozen branch prototype vector; obtaining the predicted remote sensing image segmentation result based on the two prototype vectors; inputting the remote sensing image into the trained remote sensing image segmentation model to obtain a remote sensing image segmentation result; calculating a loss function based on the remote sensing image segmentation result and a true value label; iteratively training the remote sensing image segmentation model according to the loss function to obtain a trained remote sensing image segmentation model; and the application can prevent overfitting.

Description

Technical Field

[0001] This application relates to the field of remote sensing image observation technology, and in particular to a remote sensing image segmentation method, apparatus, storage medium and electronic device based on visual Transformer. Background Technology

[0002] In the field of Earth observation, semantic segmentation of remote sensing images using deep learning technology has become an accurate and scalable method that can be used to monitor the Earth's surface and support applications such as land cover mapping, development surveys, and disaster damage assessment.

[0003] However, deep learning-based methods experience a significant performance drop when data labeling is insufficient or when dealing with unknown categories. Even with additional data, pixel-by-pixel labeling of large datasets remains an expensive and time-consuming task, especially for high-resolution remote sensing images. To address the semantic segmentation problem under low-data conditions, researchers have proposed the few-shot segmentation (FSS) method, which has made some progress in the field of natural images. Some studies have also begun to adapt and optimize FSS techniques for Earth observation missions.

[0004] Specifically, there is some debate surrounding whether the backbone network should be frozen. Some FSS methods fix the backbone network parameters to preserve pre-trained knowledge, but this may fail to capture complex feature representations specific to a particular domain. Other methods allow the backbone network to learn specific feature representations, but may carry the risk of overfitting. Summary of the Invention

[0005] This application provides a remote sensing image segmentation method, apparatus, storage medium, and electronic device based on visual Transformer, which can prevent overfitting in the process of obtaining remote sensing image segmentation results.

[0006] This application provides a remote sensing image segmentation method based on visual Transformer, including: Obtain a remote sensing image dataset, which includes query remote sensing images and supporting remote sensing images, and label the supporting remote sensing images with ground truth labels; The remote sensing dataset is input into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result. Based on the remote sensing image segmentation result and the ground truth label, the loss function is calculated. The remote sensing image segmentation model is iteratively trained according to the loss function to obtain the trained remote sensing image segmentation model. The remote sensing image segmentation model includes adjustable branches and frozen branches. The step of inputting the remote sensing dataset into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result includes: The query remote sensing image and the supporting remote sensing image are input into the adjustable branch for processing to obtain the adjustable branch prototype vector; The query remote sensing image and the supporting remote sensing image are input into the frozen branch for processing to obtain the frozen branch prototype vector. The predicted remote sensing image segmentation result is obtained based on the adjustable branch prototype vector and the frozen branch prototype vector. The remote sensing image is input into the trained remote sensing image segmentation model to obtain the remote sensing image segmentation result.

[0007] Furthermore, in the remote sensing image segmentation method based on Visual Transformer, the adjustable branch includes a first ViT module, a refiner, and a first cosine similarity classifier; the ground truth labels include supporting ground truth labels for remote sensing images and querying ground truth labels for remote sensing images. The step of inputting the query remote sensing image and the supporting remote sensing image into the adjustable branch for processing to obtain the adjustable branch prototype vector includes: The query remote sensing image and the supporting remote sensing image are input into the ViT module for preliminary feature extraction, and then optimized by the refiner to obtain the first query image features and the first supporting image features. The first supporting image features and the ground truth labels of the supporting remote sensing images are subjected to masked mean pooling to obtain an adjustable branch prototype vector.

[0008] Furthermore, in the remote sensing image segmentation method based on visual Transformer, after the step of obtaining the adjustable branch prototype vector, the method includes: The query remote sensing image is enhanced by n angular rotations to obtain n rotated and enhanced query images.

[0009] Furthermore, in the remote sensing image segmentation method based on visual Transformer, the frozen branch includes a second ViT module and a second cosine similarity classifier; The step of inputting the queried remote sensing image and the supporting remote sensing image into the frozen branch for processing to obtain the frozen branch prototype vector includes: The n rotated and enhanced query images and the supporting remote sensing images are input into the second ViT module for feature extraction to obtain the second query image features and the second supporting image features. The second supporting image features and the ground truth labels of the supporting remote sensing images are subjected to masked mean pooling to obtain the first frozen branch prototype vector. The first frozen branch prototype vector and the second query image features are input into the second cosine similarity classifier to obtain the query image pseudo-label; A second frozen branch prototype vector is generated based on the pseudo-label of the query image and the features of the second query image.

[0010] Furthermore, in the remote sensing image segmentation method based on visual Transformer, the step of obtaining the predicted remote sensing image segmentation result based on the adjustable branch prototype vector and the frozen branch prototype vector includes: The adjustable branch prototype vector is fused with the frozen branch prototype vector to obtain a fused prototype vector set; The fused prototype vector set and the features of the first query image are input into the first cosine similarity classifier to obtain the predicted remote sensing image segmentation result.

[0011] Furthermore, the remote sensing image segmentation method based on visual Transformer further includes: A weighted map is obtained based on the pixel distribution of the remote sensing query image; The loss function is calculated based on the weight map, the ground truth labels of the remote sensing image, and the predicted remote sensing image segmentation results.

[0012] Furthermore, in the remote sensing image segmentation method based on visual Transformer, the step of obtaining a weight map based on the pixel distribution of the remote sensing query image includes: The weighted graph is calculated using a first formula, which is:

[0013] in, It refers to the distribution of boundary pixels in the query image. Is it a Boolean operation? This is a Euclidean distance transformation. As the attenuation factor, It is an exponential function; The calculation of the loss function based on the weight map, the ground truth labels of the queried remote sensing image, and the predicted remote sensing image segmentation result includes: The loss function is calculated using a second formula, which is:

[0014] in, To query the ground truth labels of remote sensing images, For the predicted remote sensing image segmentation results, For loss function, They represent The coordinates are The value at that location, , These represent the width and height of the predicted remote sensing image segmentation result, respectively.

[0015] This application also provides a remote sensing image segmentation device based on a visual Transformer, including: The acquisition module is used to acquire a remote sensing image dataset, which includes query remote sensing images and supporting remote sensing images, and to annotate the supporting remote sensing images with ground truth labels. The prediction module is used to input the remote sensing dataset into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result, calculate the loss function based on the remote sensing image segmentation result and the ground truth label, and iteratively train the remote sensing image segmentation model according to the loss function to obtain the trained remote sensing image segmentation model; wherein, the remote sensing image segmentation model includes adjustable branches and frozen branches. The step of inputting the remote sensing dataset into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result includes: The query remote sensing image and the supporting remote sensing image are input into the adjustable branch for processing to obtain the adjustable branch prototype vector; The query remote sensing image and the supporting remote sensing image are input into the frozen branch for processing to obtain the frozen branch prototype vector. The predicted remote sensing image segmentation result is obtained based on the adjustable branch prototype vector and the frozen branch prototype vector. The remote sensing image is input into the trained remote sensing image segmentation model to obtain the remote sensing image segmentation result.

[0016] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described visual Transformer-based remote sensing image segmentation methods.

[0017] This application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used for steps in the visual Transformer-based remote sensing image segmentation method described above.

[0018] This application provides a remote sensing image segmentation method, apparatus, storage medium, and electronic device based on visual Transformer. This application proposes adjustable branches and frozen branches, and obtains the prototype vectors of the adjustable branches and frozen branches respectively. The remote sensing image segmentation result is obtained based on the prototype vectors of the adjustable branches and frozen branches. This application retains pre-training knowledge through the frozen branch, which can make fuller use of existing natural image datasets for large-scale pre-training, so that the remote sensing image segmentation model can learn the representations specific to the remote sensing field while preventing overfitting. Attached Figure Description

[0019] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0020] Figure 1 A flowchart of a remote sensing image segmentation method based on visual Transformer provided in an embodiment of this application.

[0021] Figure 2 Another flowchart of the remote sensing image segmentation method based on visual Transformer provided in the embodiments of this application.

[0022] Figure 3 This is a schematic diagram of the remote sensing image segmentation results provided in an embodiment of this application.

[0023] Figure 4 A schematic diagram of the structure of the remote sensing image segmentation device based on visual Transformer provided in the embodiments of this application.

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0025] Figure 6 Another structural schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] Currently, existing remote sensing image segmentation methods include PANet, a prototype alignment network designed to address few-shot segmentation problems. PANet employs a simple yet effective method to learn prototype representations of different foreground objects and backgrounds. This technique first embeds different foreground objects and backgrounds into different prototypes using a shared feature extractor, ensuring each prototype represents a class and is sufficiently distinguishable from others. Then, the query image is labeled based on the class-specific prototype with the nearest pixel distance. Even with only one support image per class, PANet provides good segmentation results. Furthermore, prototype alignment regularization is achieved by forming a new support set using the query image and its predicted mask, and performing segmentation on the original support set. This helps maintain consistency between prototypes generated from the query and those generated from the support. This technique uses a VGG-16 network as its backbone. However, it does not employ an advanced visual Transformer as its backbone. While fixing the backbone parameters preserves pre-trained knowledge, it fails to capture the complex representations of the remote sensing domain.

[0028] Currently, existing remote sensing image segmentation methods include SDM, a scale-aware few-shot semantic segmentation network capable of handling significant differences in the appearance and scale of semantic objects. This technique first utilizes a learnable backbone network (such as VGG-16 or ResNet-50) as a feature extractor to generate query and supporting features. Then, multiple prototypes are obtained using prototype mixture models (PMMs). A detailed matching module is designed to perform pixel-level matching between query features and supporting prototypes to provide more accurate parsing guidance. The fused features are input into a designed convolutional layer via spatial pyramid pooling to predict the final parsing result. In the detailed matching module, a Gaussian mixture model is first constructed and iteratively updated using an expectation-maximization algorithm to generate multiple prototypes. Then, the cosine similarity between each prototype and each element in the query features is calculated, and a pixel-level similarity matrix is ​​obtained through an argmax operation, providing accurate global matching features. The global matching features are used in subsequent parsing processes, while the global similarity map is also used to further enhance parsing guidance. To overcome scale imbalance, a scale-aware focus loss method is designed, focusing attention on small targets and evaluating hard samples through scale scores. However, it does not use an advanced visual Transformer as the backbone network, which allows the backbone network to learn specific feature representations, but carries the risk of overfitting.

[0029] To address the aforementioned problems, embodiments of this application provide a remote sensing image segmentation method, apparatus, storage medium, and electronic device based on visual Transformer. The remote sensing image segmentation apparatus based on visual Transformer provided in this application can be integrated into an electronic device, which may be a terminal, server, or other device. The terminal may include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0030] Please see Figure 1 and Figure 2 , Figure 1 A flowchart illustrating the remote sensing image segmentation method based on visual Transformer provided in this application embodiment. Figure 2 Another flowchart of the remote sensing image segmentation method based on visual Transformer provided in this application embodiment, which is applied in an electronic device, includes the following steps: S1. Obtain the remote sensing image dataset, which includes the query remote sensing images and supporting remote sensing images, and label the supporting remote sensing images with ground truth labels.

[0031] S2, input the remote sensing dataset into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result, calculate the loss function based on the remote sensing image segmentation result and ground truth label, and iteratively train the remote sensing image segmentation model according to the loss function to obtain the trained remote sensing image segmentation model; wherein, the remote sensing image segmentation model includes adjustable branches and frozen branches.

[0032] Specifically, the remote sensing image segmentation model comprises two parallel ViT branches: In the adjustable branch, ViT and the proposed refiner module are optimized through meta-learning to establish a well-structured embedding space. This branch generates pixel embeddings for remote sensing image queries and prototype vectors based on the support set (a small number of labeled samples) to obtain intra-domain representations of remote sensing image pixels and the categories to be segmented. In the frozen branch, ViT parameters pre-trained on a natural image dataset are fixed and used solely as a powerful feature extractor to generate prototypes for the support and query sets. Finally, the prototypes from both branches are explicitly fused through a prototype fusion strategy to obtain the final prediction. The entire model is supervised by a weighted cross-entropy loss, which emphasizes boundary regions. This dual-branch strategy aims to synergistically integrate intra-domain representations and pre-trained model guidance.

[0033] The process of inputting the remote sensing dataset into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation results includes: S21, input the query remote sensing image and the supporting remote sensing image into the adjustable branch for processing to obtain the adjustable branch prototype vector.

[0034] Specifically, the adjustable branches include a first ViT (Vision Transformer) module, a refiner, and a first cosine similarity classifier. The ViT module is pre-trained on a natural image dataset (typically ImageNet-1k). The refiner comprises two parallel branches: one branch uses a deconvolution LayerNorm, GELU activation function, and a 1x1 concatenation to double the resolution; the other branch is a bilinear interpolation module that preserves the original feature information. The refiner module enhances the resolution and representation of the original feature map through these operations, supporting subsequent segmentation tasks.

[0035] In one embodiment, step S21 includes the following steps: S211, the query remote sensing image and the supporting remote sensing image are input into the ViT module for preliminary feature extraction, and then optimized by the refiner to obtain the first query image features and the first supporting image features.

[0036] Specifically, querying remote sensing images With support for remote sensing images The data is fed into the ViT module in the adjustable branch for initial feature extraction. and The data is then sent to the refiner module for optimization to obtain the first query image features. and first supporting image features This process can be expressed by the formula:

[0037]

[0038]

[0039]

[0040] The refiner module consists of and composition: This means that the resolution is doubled first through a deconvolution, while the channel dimension is reduced to half of the original. After LayerNorm and GELU activation, a 1x1 convolution is used to restore the original number of channels. This represents double bilinear interpolation, used to improve resolution while preserving the original information. The parallel connection forms a residual structure, which is composed of Responsible for learning gain information.

[0041] S212, perform masked mean pooling on the first supporting image features and the ground truth labels of the supporting remote sensing images to obtain the adjustable branch prototype vector.

[0042] Specifically, the first supporting image features Support for truth labels on remote sensing images The adjustable branch prototype vector is obtained by mask mean pooling. .

[0043] Furthermore, considering the rotation invariance of remote sensing images, n angular rotation enhancements are applied to the query image, with each rotation angle being [value missing]. This yields n rotated and enhanced query images. For example, n can be set to 4.

[0044] S22, input the query remote sensing image and the supporting remote sensing image into the frozen branch for processing, and obtain the frozen branch prototype vector.

[0045] In one embodiment, the frozen branch includes a second ViT module and a second cosine similarity classifier, and step S22 includes the following steps: S221, input the n rotated and enhanced query images and supporting remote sensing images into the second ViT module for feature extraction to obtain the second query image features and the second supporting image features.

[0046] Specifically, the rotated enhanced query image, along with the supporting remote sensing image, is fed into the ViT module of the frozen branch to extract features and obtain the corresponding second query image features. Second Support Image Features .

[0047] S222, perform masked mean pooling on the second supporting image features and the ground truth labels of the supporting remote sensing images to obtain the prototype vector of the first frozen branch.

[0048] S223, input the prototype vector of the first frozen branch and the features of the second query image into the second cosine similarity classifier to obtain the pseudo-label of the query image.

[0049] S224, Generate the prototype vector of the second frozen branch based on the pseudo-label of the query image and the features of the second query image.

[0050] Specifically, mask mean pooling is used based on the second support image features. Support for truth labels on remote sensing images Calculate the prototype vector of the first frozen branch The prototype vector of the first frozen branch is compared with the features of the second query image. Feed into cosine similarity classifier In the process of calculating the cosine similarity between the prototype vector of the first frozen branch and the features of the second query image one by one, and assigning the label corresponding to the prototype vector of the first frozen branch based on the largest cosine similarity, pseudo-labels for the query image are obtained. Based on the pseudo-labels of the query image and the features of the second query image, a frozen branch prototype vector is generated. The formula for this process is as follows:

[0051]

[0052] in, This indicates mask mean pooling.

[0053] S23, the predicted remote sensing image segmentation result is obtained based on the adjustable branch prototype vector and the frozen branch prototype vector.

[0054] In one embodiment, step S23 includes the following steps: S231, merge the adjustable branch prototype vector with the frozen branch prototype vector to obtain the fused prototype vector set.

[0055] Specifically, the prototype vectors of the adjustable branch from the adjustable branch and the prototype vectors of the second frozen branch from the frozen branch are fused, integrating the in-domain representation of the adjustable branch and the pre-trained representation of the frozen branch. By combining the advantages of both, the overfitting risk of the adjustable branch is avoided, thereby achieving better segmentation results. Specifically, the two prototype vectors are fused using a linear weighted average to obtain a fused prototype set. The formula is as follows:

[0056] Here, α and β are hyperparameters, usually set to 0.7 and 0.3.

[0057] S232, the fused prototype vector set and the features of the first query image are input into the first cosine similarity classifier to obtain the predicted remote sensing image segmentation result, which can be expressed by the following formula:

[0058] Furthermore, the loss function of the remote sensing image segmentation model can be calculated through the following steps: SA is a weighted map derived from the pixel distribution of a remotely sensed query image.

[0059] The weighted graph can be calculated using the first formula, which is:

[0060] in, It refers to the distribution of boundary pixels in the query image. Is it a Boolean operation? This is a Euclidean distance transformation. This is the attenuation factor (which can be set to 5). It is an exponential function.

[0061] SB calculates the loss function based on the weighted graph, the ground truth labels of the remote sensing images, and the predicted remote sensing image segmentation results.

[0062] The loss function can be calculated using the second formula, which is:

[0063] in, To query the ground truth labels of remote sensing images, For the predicted remote sensing image segmentation results, For loss function, They represent The coordinates are The value at that location, , These represent the width and height of the predicted remote sensing image segmentation result, respectively.

[0064] S24. Input the remote sensing image into the trained remote sensing image segmentation model to obtain the remote sensing image segmentation result. Figure 3 This is a schematic diagram of the remote sensing image segmentation results provided in the embodiments of this application. (Refer to...) Figure 3 .

[0065] This application distinguishes between query remote sensing images and supporting remote sensing images, predicting remote sensing image segmentation results by processing both types of images. This eliminates the need for extensive remote sensing image data annotation, reducing manual annotation costs. The application proposes a frozen branch strategy, preserving pre-trained knowledge and making fuller use of existing natural image datasets for large-scale pre-training. This allows the model to learn remote sensing-specific representations while preventing overfitting. Although the weights of the frozen branch are not updated, they play a role in learning domain representations through prototype regularization, helping to guide and promote the construction of a well-structured embedding space. Furthermore, this application adapts the advanced architecture from natural image tasks—the Visual Transformer—to remote sensing image segmentation under small sample conditions in a simple and efficient manner, improving resource utilization and segmentation accuracy.

[0066] Based on the method described in the above embodiments, this embodiment will further describe it from the perspective of a remote sensing image segmentation device based on visual Transformer. The remote sensing image segmentation device based on visual Transformer can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other devices. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0067] Please see Figure 4 , Figure 4 This application provides a specific description of a remote sensing image segmentation device based on a visual Transformer, which is applied in electronic devices. The device may include: The acquisition module is used to acquire a remote sensing image dataset, which includes query remote sensing images and supporting remote sensing images, and to annotate the supporting remote sensing images with ground truth labels. The prediction module is used to input the remote sensing dataset into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation results. Based on the remote sensing image segmentation results and ground truth labels, the loss function is calculated, and the remote sensing image segmentation model is iteratively trained according to the loss function to obtain the trained remote sensing image segmentation model. The remote sensing image segmentation model includes adjustable branches and frozen branches. The remote sensing dataset is input into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation results, including: The query remote sensing image and the supporting remote sensing image are input into the adjustable branch for processing to obtain the prototype vector of the adjustable branch. The query remote sensing image and the supporting remote sensing image are input into the frozen branch for processing to obtain the frozen branch prototype vector. The predicted remote sensing image segmentation results are obtained based on the adjustable branch prototype vector and the frozen branch prototype vector. The remote sensing image is input into the trained remote sensing image segmentation model to obtain the remote sensing image segmentation result.

[0068] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0069] In addition, embodiments of this application also provide an electronic device, which may be a computer, tablet computer, or other similar device. Figure 5As shown, the electronic device 400 includes a processor 401 and a memory 402. The processor 401 and the memory 402 are electrically connected.

[0070] The processor 401 is the control center of the electronic device 400. It connects various parts of the electronic device through various interfaces and lines. By running or loading the application program stored in the memory 402 and calling the data stored in the memory 402, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.

[0071] In this embodiment, the processor 401 in the electronic device 400 loads the instructions corresponding to the processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 runs the application programs stored in the memory 402 to realize various functions: Obtain a remote sensing image dataset, which includes query remote sensing images and supporting remote sensing images, and label the supporting remote sensing images with ground truth labels; The remote sensing dataset is input into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result. Based on the remote sensing image segmentation result and the ground truth label, the loss function is calculated. The remote sensing image segmentation model is iteratively trained according to the loss function to obtain the trained remote sensing image segmentation model. The remote sensing image segmentation model includes adjustable branches and frozen branches. The step of inputting the remote sensing dataset into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result includes: The query remote sensing image and the supporting remote sensing image are input into the adjustable branch for processing to obtain the adjustable branch prototype vector; The query remote sensing image and the supporting remote sensing image are input into the frozen branch for processing to obtain the frozen branch prototype vector. The predicted remote sensing image segmentation result is obtained based on the adjustable branch prototype vector and the frozen branch prototype vector. The remote sensing image is input into the trained remote sensing image segmentation model to obtain the remote sensing image segmentation result.

[0072] This electronic device can implement the steps of any embodiment of the remote sensing image segmentation method based on visual Transformer provided in the embodiments of this application. Therefore, it can achieve the beneficial effects that any remote sensing image segmentation method based on visual Transformer provided in the embodiments of this invention can achieve, as detailed in the preceding embodiments, which will not be repeated here.

[0073] Figure 6A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the remote sensing image segmentation method based on visual Transformer provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0074] RF circuit 510 is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals and vice versa, thereby enabling communication with communication networks or other devices. RF circuit 510 may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, subscriber identity modules (SIM cards), memory, etc. RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). The aforementioned wireless networks may use various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messages, and any other suitable communication protocols, including those that have not yet been developed.

[0075] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, such as taking pictures with the front-facing camera, processing the captured images, and switching the display colors of the content displayed on the screen. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0076] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control. Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0077] Audio circuitry 560, speaker 561, and microphone 562 provide an audio interface between the user and electronic device 500. Audio circuitry 560 converts received audio data into electrical signals and transmits them to speaker 561, where speaker 561 converts them into sound signals for output. Conversely, microphone 562 converts collected sound signals into electrical signals, which are then received by audio circuitry 560, converted back into audio data, and processed by processor 580. The audio data is then transmitted via RF circuitry 510 to, for example, another terminal, or output to memory 520 for further processing. Audio circuitry 560 may also include an earphone jack to facilitate communication between external headphones and electronic device 500.

[0078] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0079] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0080] Electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 590 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0081] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations: Obtain a remote sensing image dataset, which includes query remote sensing images and supporting remote sensing images, and label the supporting remote sensing images with ground truth labels; The remote sensing dataset is input into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result. Based on the remote sensing image segmentation result and the ground truth label, the loss function is calculated. The remote sensing image segmentation model is iteratively trained according to the loss function to obtain the trained remote sensing image segmentation model. The remote sensing image segmentation model includes adjustable branches and frozen branches. The step of inputting the remote sensing dataset into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result includes: The query remote sensing image and the supporting remote sensing image are input into the adjustable branch for processing to obtain the adjustable branch prototype vector; The query remote sensing image and the supporting remote sensing image are input into the frozen branch for processing to obtain the frozen branch prototype vector. The predicted remote sensing image segmentation result is obtained based on the adjustable branch prototype vector and the frozen branch prototype vector. The remote sensing image is input into the trained remote sensing image segmentation model to obtain the remote sensing image segmentation result.

[0082] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0083] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the visual Transformer-based remote sensing image segmentation method provided by the present invention.

[0084] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0085] Since the instructions stored in the storage medium can execute the steps in any embodiment of the remote sensing image segmentation method based on visual Transformer provided in the embodiments of the present invention, the beneficial effects that any remote sensing image segmentation method based on visual Transformer provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0086] The foregoing has provided a detailed description of a remote sensing image segmentation method, apparatus, storage medium, and electronic device based on a visual Transformer, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A remote sensing image segmentation method based on visual Transformer, characterized in that, The method includes: Obtain a remote sensing image dataset, which includes query remote sensing images and supporting remote sensing images, and label the supporting remote sensing images with ground truth labels; The remote sensing image dataset is input into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result. The loss function is calculated based on the remote sensing image segmentation result and the ground truth label. The remote sensing image segmentation model is iteratively trained according to the loss function to obtain the trained remote sensing image segmentation model. The remote sensing image segmentation model includes adjustable branches and frozen branches; the remote sensing image dataset is input into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result, including: The query remote sensing image and the supporting remote sensing image are input into the adjustable branch for processing to obtain an adjustable branch prototype vector. The adjustable branch includes a first ViT module, a refiner, and a first cosine similarity classifier. The ground truth labels include ground truth labels for the supporting remote sensing image and ground truth labels for the query remote sensing image. The process of inputting the query remote sensing image and the supporting remote sensing image into the adjustable branch to obtain the adjustable branch prototype vector includes: inputting the query remote sensing image and the supporting remote sensing image into the ViT module for preliminary feature extraction, and optimizing them through the refiner to obtain first query image features and first supporting image features; and performing masked mean pooling on the first supporting image features and the ground truth labels for the supporting remote sensing image to obtain the adjustable branch prototype vector. The query remote sensing image and the supporting remote sensing image are input into the frozen branch for processing to obtain a frozen branch prototype vector. The frozen branch includes a second ViT module and a second cosine similarity classifier. The process of inputting the query remote sensing image and the supporting remote sensing image into the frozen branch to obtain the frozen branch prototype vector includes: inputting n rotated and enhanced query remote sensing images and the supporting remote sensing images into the second ViT module for feature extraction to obtain second query image features and second supporting image features; performing masked mean pooling on the second supporting image features and the ground truth labels of the supporting remote sensing images to obtain a first frozen branch prototype vector; inputting the first frozen branch prototype vector and the second query image features into the second cosine similarity classifier to obtain query image pseudo-labels; and generating a second frozen branch prototype vector based on the query image pseudo-labels and the second query image features. The predicted remote sensing image segmentation result is obtained based on the adjustable branch prototype vector and the frozen branch prototype vector. A weighted map is obtained based on the pixel distribution of the queried remote sensing image; The loss function is calculated based on the weight map, the ground truth labels of the remote sensing images, and the predicted remote sensing image segmentation results; The process of obtaining a weight map based on the pixel distribution of the queried remote sensing image includes: calculating the weight map using a first formula, wherein the first formula is: in, It refers to the distribution of boundary pixels in the query image. Is it a Boolean operation? This is a Euclidean distance transformation. As the attenuation factor, It is an exponential function; The loss function is calculated based on the weight map, the ground truth labels of the queried remote sensing images, and the predicted remote sensing image segmentation results, including: calculating the loss function using a second formula, wherein the second formula is: in, For loss function, Representing the weighted graphs respectively Query the true value labels of remote sensing images Predicted remote sensing image segmentation results At coordinates The value at that location, , These are the width and height of the predicted remote sensing image segmentation result, respectively; The remote sensing image is input into the trained remote sensing image segmentation model to obtain the remote sensing image segmentation result.

2. The remote sensing image segmentation method based on visual Transformer according to claim 1, characterized in that, After obtaining the adjustable branch prototype vector, the following steps are included: The query remote sensing image is enhanced by n angular rotations to obtain n rotated and enhanced query remote sensing images.

3. The remote sensing image segmentation method based on visual Transformer according to claim 1, characterized in that, The predicted remote sensing image segmentation result obtained based on the adjustable branch prototype vector and the frozen branch prototype vector includes: The adjustable branch prototype vector is fused with the frozen branch prototype vector to obtain a fused prototype vector set; The fused prototype vector set and the features of the first query image are input into the first cosine similarity classifier to obtain the predicted remote sensing image segmentation result.

4. A remote sensing image segmentation device based on visual Transformer, wherein the remote sensing image segmentation device based on visual Transformer is used to implement the remote sensing image segmentation method based on visual Transformer as described in claim 1, characterized in that, include: The acquisition module is used to acquire a remote sensing image dataset, which includes query remote sensing images and supporting remote sensing images, and to annotate the supporting remote sensing images with ground truth labels. The prediction module is used to input the remote sensing image dataset into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result, calculate the loss function based on the remote sensing image segmentation result and the ground truth label, and iteratively train the remote sensing image segmentation model according to the loss function to obtain the trained remote sensing image segmentation model; wherein, the remote sensing image segmentation model includes adjustable branches and frozen branches. The step of inputting the remote sensing image dataset into the remote sensing image segmentation model to obtain the predicted remote sensing image segmentation result includes: The query remote sensing image and the supporting remote sensing image are input into the adjustable branch for processing to obtain the adjustable branch prototype vector; The query remote sensing image and the supporting remote sensing image are input into the frozen branch for processing to obtain the frozen branch prototype vector. The predicted remote sensing image segmentation result is obtained based on the adjustable branch prototype vector and the frozen branch prototype vector. The remote sensing image is input into the trained remote sensing image segmentation model to obtain the remote sensing image segmentation result.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the remote sensing image segmentation method based on visual Transformer as described in any one of claims 1 to 3.

6. An electronic device, characterized in that, The method includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps in the remote sensing image segmentation method based on visual Transformer as described in any one of claims 1 to 3.

Images