Interactive segmentation method for remote sensing video moving target

Abstract

The invention belongs to the technical field of remote sensing image processing, and particularly discloses an interactive segmentation method for a remote sensing video moving target, and the method comprises the steps: carrying out the training of a segmentation model through a data set, and obtaining a trained segmentation model, the segmentation model comprises a space-time enhancement image encoder, a prompt encoder, a mask decoder and a decoupling high-frequency compensation decoder; inputting a to-be-segmented remote sensing image or video into the trained segmentation model to obtain a segmentation result; the space-time enhanced image encoder comprises a low-rank adaptation matrix and a time sequence memory link and is used for extracting features of an input video image; the prompt encoder is used for converting an interaction prompt input by a user into a high-dimensional prompt vector; the mask decoder is used for obtaining a preliminary segmentation result based on the features of the input video image and the high-dimensional prompt vector; and the decoupling high-frequency compensation decoder is used for extracting shallow high-frequency features from the features extracted by the space-time enhanced image encoder, and optimizing the preliminary segmentation result based on the shallow high-frequency features.

Description

Technical Field

[0001] This application belongs to the field of remote sensing image processing technology, and more specifically, relates to an interactive segmentation method for moving targets in remote sensing video. Background Technology

[0002] With the widespread application of remote sensing video and high-resolution aerial imagery in fields such as national defense monitoring, disaster assessment, and traffic management, accurate, fast, and interactive segmentation of moving targets has become a core technical requirement. Although general-purpose large models, represented by the Segment Anything Model (SAM), have achieved breakthroughs in the field of natural images, they still reveal the following core limitations when directly transferred to remote sensing video scenes: 1. Severe loss of spatial features of small targets: SAM uses a fixed image input resolution. In remote sensing images, targets such as vehicles and ships often occupy only a few pixels. After multiple levels of downsampling by the encoder, the semantic information of these small targets is easily dissipated, leading to missed detections or segmentation holes; 2. Lack of temporal continuity: Most existing improvement schemes are based on static single-frame processing. Frequent cloud and fog obstructions, rapid changes in light and shadow, and rapid target movement in remote sensing videos can cause severe flickering or target loss in the segmentation mask between frames; 3. Mismatch of domain features and blurred boundaries: Remote sensing targets have characteristics such as arbitrary orientation and highly similar textures (e.g., camouflage, background disguise). When processing such high-altitude images, general models have difficulty focusing their attention, and the generated mask boundaries often exhibit jagged edges or broken topological structures under complex background interference; 4. The contradiction between adaptation cost and generalization ability: Although full parameter fine-tuning can improve performance, remote sensing data annotation is costly and the model size is huge. Full training not only consumes massive computing power, but also makes it easier for the model to lose its original zero-sample generalization advantage and become unable to cope with unseen sensor types.

[0003] In summary, existing remote sensing video segmentation methods have low segmentation accuracy. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application aims to provide an interactive segmentation method for moving targets in remote sensing videos, which addresses the low segmentation accuracy caused by severe loss of spatial features of small targets, lack of temporal continuity, and mismatch and blurred boundaries in existing remote sensing video segmentation methods.

[0005] To achieve the above objectives, in a first aspect, this application provides an interactive segmentation method for moving targets in remotely sensed video, comprising: The segmentation model is trained using a pre-acquired dataset to obtain a trained segmentation model, which includes a spatiotemporal enhancement image encoder, a cue encoder, a mask decoder, and a decoupled high-frequency compensation decoder. Input the remote sensing image or video to be segmented into the trained segmentation model to obtain the segmentation result; The spatiotemporal enhanced image encoder includes a low-rank adaptation matrix and a temporal memory link, used to extract features from the input video image; The prompt encoder is used to convert user-input interactive prompts into high-dimensional prompt vectors; The mask decoder is used to obtain preliminary segmentation results based on the features of the input video image and the high-dimensional cue vector; The decoupled high-frequency compensation decoder is used to extract shallow high-frequency features from the features extracted by the spatiotemporal enhancement image encoder, and to optimize the preliminary segmentation results based on the shallow high-frequency features.

[0006] This application constructs and trains a segmentation model that introduces a low-rank adaptation matrix (LoRA) and a temporal memory link into the image encoder, enabling the static encoder to process video stream features and completing cross-domain transfer from general natural images to the field of high-altitude remote sensing. This effectively solves the problem of segmentation fragmentation caused by occlusion, fog, or shadows in remote sensing videos. By decoupling the high-frequency compensation decoder and laterally connecting it, the model extracts early texture features (high-frequency information) from the image encoder, compensating for the loss of spatial resolution in deep semantic features. Geometric constraints are applied to the blurred boundaries of remote sensing targets, making the mask edges fit the geometric contours of the remote sensing targets more closely, significantly improving boundary integrity, and thus improving segmentation accuracy.

[0007] According to the interactive segmentation method for moving targets in remote sensing video provided in this application, the training process of the spatiotemporal augmentation image encoder includes: The pre-trained Vision Transformer (ViT) is used as an image encoder, pre-trained weights are loaded, and the backbone parameters are frozen. A low-rank adaptation matrix is ​​injected into the self-attention projection layer of ViT, and the low-rank adaptation matrix is ​​used to capture the texture structure and scale changes of the image. A temporal memory link is constructed to process the encoded features of the (t-1)th frame of the video, generate a temporal bias term, and inject the temporal bias term into the self-attention calculation of the tth frame of the video, where t is a positive integer greater than 1 and less than or equal to the total frame length of the video. Based on the pre-acquired dataset, the low-rank adaptation matrix and the temporal memory link are trained; Unfreeze and fine-tune the last layer of the ViT Transformer Block, injecting remote sensing-specific high-level semantic knowledge.

[0008] This application injects spatial and temporal (ST)-LoRA branches in parallel into the self-attention projection layer of the ViT encoder and introduces a temporal memory link, enabling the model to utilize the context of previous video frames to effectively solve the segmentation fragmentation problem caused by target occlusion, fog, or shadows in remote sensing videos. By fine-tuning the last Transformer Block and associating it with features from previous frames, the model not only possesses the discriminative power of remote sensing domain features but also effectively handles the problems of target occlusion and instantaneous disappearance in videos. This enhances the ability to suppress "false targets" in complex remote sensing backgrounds, significantly reduces the mixing rate when multiple targets are near-field interference, and improves segmentation accuracy.

[0009] According to the interactive segmentation method for moving targets in remote sensing video provided in this application, the training process of the mask decoder and the decoupled high-frequency compensation decoder includes: Intermediate feature maps are extracted from the shallow layer of the spatiotemporal enhanced image encoder; The intermediate feature map is subjected to channel dimensionality reduction so that the dimensionality-reduced intermediate feature map is aligned with the upsampled features of the mask decoder in the channel dimension; The reduced intermediate feature map is fused with the semantic features of the mask decoder to form a decoupled high-frequency compensation path; During training, the parameters of the mask decoder and the parameters of the decoupled high-frequency compensation decoder are updated alternately.

[0010] This application compensates for the spatial resolution loss of deep semantic features by fusing early texture features with later semantic context, making the mask edges more closely fit the geometric contours of remote sensing targets. It fuses the dimensionality-reduced intermediate feature maps with the semantic features of the mask decoder to form a decoupled high-frequency compensation path, which is used to repair problems such as blurred boundaries and missing details. It adopts a strategy of alternating optimization with the original mask decoder to ensure that the model does not destroy the large-scale pre-trained spatial priors accumulated by the original SAM while learning "how to generate high-quality masks", thus ensuring stability when dealing with unseen remote sensing scenes. It specifically applies geometric constraints to the blurred boundaries of remote sensing targets, significantly improving boundary integrity and segmentation accuracy.

[0011] According to the interactive segmentation method for moving targets in remote sensing video provided in this application, the inference process of the trained segmentation model includes: Based on the user-input target area-related prompts and video images, determine the center point of the target area in the video image; The video image is cropped with the center point of the target area as the center to obtain a local image block; Upsample the local image patch; The upsampled local image patch is input into the trained segmentation model to obtain a high-resolution mask; The high-resolution mask is mapped back to the coordinate system of the original video image to generate pixel-level segmentation results.

[0012] This application performs adaptive Region of Interest (RoI) cropping and upsampling based on user input during the model inference stage. It can dynamically adjust the cropping window and perform optimal resampling based on the preliminary prediction, mapping extremely small targets (such as vehicles and small ships) back to the high-resolution feature space. This greatly enhances the pixel-level resolution of extremely small targets, completely avoids the loss of information under standard input size through downsampling, and improves segmentation accuracy.

[0013] Secondly, this application provides an interactive segmentation apparatus for moving targets in remotely sensed video, comprising: The training module is used to train the segmentation model using a pre-acquired dataset to obtain a trained segmentation model, which includes a spatiotemporal enhancement image encoder, a cue encoder, a mask decoder, and a decoupled high-frequency compensation decoder. The segmentation module is used to input the remote sensing image or video to be segmented into the trained segmentation model and obtain the segmentation result. The spatiotemporal enhanced image encoder includes a low-rank adaptation matrix and a temporal memory link, used to extract features from the input video image; The prompt encoder is used to convert user-input interactive prompts into high-dimensional prompt vectors; The mask decoder is used to obtain preliminary segmentation results based on the features of the input video image and the high-dimensional cue vector; The decoupled high-frequency compensation decoder is used to extract shallow high-frequency features from the features extracted by the spatiotemporal enhancement image encoder, and to optimize the preliminary segmentation results based on the shallow high-frequency features.

[0014] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the interactive segmentation method for a moving target in remotely sensed video as described in the first aspect or any possible implementation thereof.

[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed on a processor, causes the processor to perform the interactive segmentation method for a moving target in remotely sensed video as described in the first aspect or any possible implementation thereof.

[0016] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to execute the interactive segmentation method for moving targets in remotely sensed video as described in the first aspect or any possible implementation of the first aspect.

[0017] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0018] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: This application constructs and trains a segmentation model that introduces a low-rank adaptation matrix and temporal memory link into the image encoder, enabling the static encoder to process video stream features and achieve cross-domain transfer from general natural images to the field of high-altitude remote sensing. This effectively solves the problem of segmentation fragmentation caused by occlusion, fog, or shadows in remote sensing videos. By decoupling the high-frequency compensation decoder and laterally connecting it, the model extracts early texture features (high-frequency information) from the image encoder, compensating for the loss of spatial resolution in deep semantic features. Geometric constraints are applied to the blurred boundaries of remote sensing targets, making the mask edges fit the geometric contours of the remote sensing targets more closely, significantly improving boundary integrity, and thus improving segmentation accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the interactive segmentation method for moving targets in remote sensing video provided in an embodiment of this application; Figure 2 This is a schematic diagram of the architecture of the segmentation model provided in the embodiments of this application; Figure 3 This is a schematic diagram of the ST-LoRA introduction method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the decoupled high-frequency compensation decoder provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the interactive segmentation device for moving targets in remote sensing video provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0023] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0024] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0025] Next, combined Figures 1-4 The interactive segmentation method for moving targets in remote sensing video provided in the embodiments of this application is introduced.

[0026] Figure 1 This is a flowchart illustrating the interactive segmentation method for moving targets in remote sensing video provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps: Step S1: Train the segmentation model using the pre-acquired dataset to obtain the trained segmentation model. The segmentation model includes a spatiotemporal enhancement image encoder, a cue encoder, a mask decoder, and a decoupled high-frequency compensation decoder. Step S2: Input the remote sensing image or video to be segmented into the trained segmentation model to obtain the segmentation result; The spatiotemporal augmented image encoder includes a low-rank adaptation matrix and a temporal memory link for extracting features from the input video image; A prompt encoder is used to transform user-input prompts into high-dimensional prompt vectors; The mask decoder is used to obtain preliminary segmentation results based on the features of the input video image and the high-dimensional cue vector; The decoupled high-frequency compensation decoder is used to extract shallow high-frequency features from the features extracted by the spatiotemporally enhanced image encoder, and to optimize the preliminary segmentation results based on the shallow high-frequency features.

[0027] The core idea of ​​this application is to enhance the fine perception and temporal stability of Remote Sensing Video Object Segmentation (RSVOS) in the remote sensing video dimension. It proposes a Remote Sensing Video Object Segmentation (RSVOS)-SAM++ architecture, whose design principles include: First, achieving extremely low-cost domain adaptation through ST-LoRA and introducing a temporal memory link into the interactive segmentation framework to solve the problem of video stream coherence; second, solving the persistent problem of blurred remote sensing target boundaries by decoupling the high-frequency decoder (DHF-Decoder) and forcibly constraining the mask geometry using low-level texture features; and finally, completely breaking the limitation of fixed resolution from the inference logic through the Ada-Inference adaptive pipeline, achieving pixel-level resolution of extremely small targets.

[0028] Optionally, a training and evaluation dataset for interactive segmentation of remote sensing videos is first constructed. The dataset includes: original remote sensing video sequences or multi-temporal remote sensing images, corresponding pixel-level segmentation annotation masks, optional interactive prompts (points, boxes, coarse masks, etc.), timestamps or frame sequence information (for temporal modeling), and metadata such as resolution and sensor type (which can be used for adaptive inference). The dataset is then divided into training, validation, and test sets.

[0029] Figure 2 This is a schematic diagram of the architecture of the segmentation model provided in the embodiments of this application, such as... Figure 2 As shown, the segmentation model constructed in this application includes the following components: Spatiotemporal Enhanced Image Encoder: Adopting the ViT architecture, which includes the ST-LoRA module, the static encoder is able to process video stream features by injecting a low-rank adapter into the self-attention layer and attaching a temporal memory link.

[0030] The prompt encoder is compatible with the original point, box, and mask interaction modes of SAM, and converts user interaction intentions into high-dimensional prompt vectors in real time.

[0031] Mask decoder: Obtains preliminary segmentation results based on the features of the input video image and high-dimensional cue vectors.

[0032] Decoupled high-frequency compensation decoder: Based on the original mask decoder, a high-frequency feature fusion path is added. By extracting the edge texture information of the shallow layer of the encoder, the mask can be reconstructed in a refined manner.

[0033] Optionally, the segmentation model can be trained end-to-end based on the training set to obtain a trained segmentation model.

[0034] Optionally, the remote sensing image or video frame to be segmented is input into the trained segmentation model, and combined with user-provided interactive prompts (such as points, boxes, or coarse masks), the segmentation result output by the segmentation model is obtained.

[0035] The interactive segmentation method for moving targets in remote sensing videos provided in this application constructs and trains a segmentation model. By introducing a low-rank adaptation matrix and temporal memory link into the image encoder, the static encoder is enabled to process video stream features, completing the cross-domain transfer from general natural images to the high-altitude remote sensing domain. This effectively solves the segmentation fragmentation problem caused by occlusion, fog, or shadows in remote sensing videos. By decoupling the high-frequency compensation decoder and laterally connecting it, the method extracts early texture features (high-frequency information) from the image encoder, compensating for the loss of spatial resolution in deep semantic features. Geometric constraints are applied to the blurred boundaries of remote sensing targets, making the mask edges fit the geometric contours of the remote sensing targets more closely, significantly improving boundary integrity, and thus improving segmentation accuracy.

[0036] In some embodiments, the training process of the spatiotemporal augmentation image encoder includes: The pre-trained ViT is used as an image encoder, pre-trained weights are loaded, and the backbone parameters are frozen. A low-rank adaptation matrix is ​​injected into the self-attention projection layer of ViT. The low-rank adaptation matrix is ​​used to capture the texture structure and scale changes of the image. A temporal memory link is constructed to process the encoded features of the (t-1)th frame of the video, generate a temporal bias term, and inject the temporal bias term into the self-attention calculation of the tth frame of the video, where t is a positive integer greater than 1 and less than or equal to the total frame length of the video. Based on the pre-acquired dataset, the low-rank adaptation matrix and temporal memory link are trained; Unfreeze and fine-tune the last layer of the ViT Transformer Block, injecting remote sensing-specific high-level semantic knowledge.

[0037] Figure 3 This is a schematic diagram of the ST-LoRA introduction method provided in the embodiments of this application, as shown below. Figure 3 As shown, in one embodiment of this application, the training process of the spatiotemporal augmentation image encoder includes: 1a. Initialize the backbone network: Use a pre-trained Vision Transformer (ViT) as the basic image encoder, load its pre-trained weights on large-scale natural images or remote sensing data, and freeze its backbone parameters. 2a, Inject ST-LoRA adaptation module: In each Transformer Block of ViT, a low-rank adaptation matrix is ​​introduced for the Query(Q) and Value(V) projection layers, with its rank set to r (e.g., r=8), to efficiently capture the texture structure and scale changes unique to the remote sensing field; 3a. Constructing a temporal memory link: To adapt to remote sensing video input, a cross-frame temporal memory unit is introduced: The encoded features of the (t-1)th frame are mapped to a temporal bias term through a lightweight gating module, and the bias term is injected into the self-attention calculation of the tth frame, so that the encoder has the ability to transmit information across frames and perform dynamic modeling. 4a, Freeze the backbone and train only LoRA and temporal memory links: In the initial stage, freeze the ViT backbone parameters and only update the LoRA and temporal memory link parameters to ensure that the model completes domain adaptation with low computational overhead; 5a, Deep Refinement: For complex backgrounds in remote sensing scenes (such as ports, forests, camouflaged targets, etc.), the last layer of the ViT Transformer Block is unfrozen and fine-tuned, injecting remote sensing-specific high-level semantic knowledge, which significantly enhances the model's ability to identify target entities in complex backgrounds. 6a. Convergence and weight saving: Once the validation set metrics stabilize, stop training, save the final encoder weights, and complete the training of the image encoder.

[0038] By training the spatiotemporal augmentation image encoder, cross-domain transfer from general natural images to the high-altitude remote sensing domain can be completed with minimal computational cost. It maintains the powerful zero-shot generalization ability of SAM. The constructed temporal memory link enables the model to utilize the context of previous video frames, effectively solving the problem of target segmentation breakage caused by occlusion, fog, or shadows in remote sensing videos. Fine-tuning the final layer update enhances the ability to suppress "false targets" in complex remote sensing backgrounds and significantly reduces the mixing rate when multiple targets are near-field interference.

[0039] In some embodiments, the training process for the mask decoder and the decoupled high-frequency compensation decoder includes: Extracting intermediate feature maps from the shallow layers of a spatiotemporally enhanced image encoder; Channel dimensionality reduction is performed on the intermediate feature map to align the dimensionality-reduced intermediate feature map with the upsampled features of the mask decoder in the channel dimension; The intermediate feature map after dimensionality reduction is fused with the semantic features of the mask decoder to form a decoupled high-frequency compensation path, which is used for repair. During training, the parameters of the mask decoder and the decoupled high-frequency compensation decoder are updated alternately.

[0040] Figure 4This is a schematic diagram of the structure of the decoupled high-frequency compensation decoder provided in the embodiments of this application, as shown below. Figure 4 As shown, in one embodiment of this application, the training process for the mask decoder and the decoupled high-frequency compensation decoder (DHF-Decoder) includes: 1b, Construct a multi-scale high-frequency feature extraction path: Extract intermediate feature maps from the shallow layers of the ViT encoder (such as the 2nd and 4th layers), which contain richer high-frequency edge and texture information; 2b, Dimension Alignment and Feature Mapping: Shallow features are reduced in channel dimension by 1×1 convolution, so that they are aligned with the upsampled features of the mask decoder in the channel dimension; 3b, High-frequency-semantic fusion: The shallow high-frequency features are concatenated or weighted with the semantic features of the mask decoder to form a decoupled high-frequency compensation path, which is used to repair problems such as blurred boundaries and missing details; 4b. The alternating optimization strategy employs an asymmetric update mechanism: in odd iterations, only the DHF-Decoder parameters are updated; in even iterations, only the mask decoder, i.e., the original SAM Decoder parameters, are updated; the encoder parameters are kept frozen or updated with a low learning rate. This strategy can prevent new modules from causing catastrophic forgetting of the prior knowledge in the original SAM space.

[0041] By training the DHF-Decoder, lateral connections can be used to extract early texture features (high-frequency information) from the encoder, compensating for the spatial resolution loss of deep semantic features and making the mask edges more closely match the geometric contours of the remote sensing target (such as building edges and wing outlines). The alternating optimization strategy ensures that the model learns "how to generate high-quality masks" without destroying the large-scale pre-trained spatial priors accumulated from the original SAM, guaranteeing stability when dealing with unseen remote sensing scenes.

[0042] In some embodiments, the inference process of the trained segmentation model includes: Based on the user-input target area-related prompts and video images, determine the center point of the target area in the video image; The video image is cropped with the center point of the target area as the center to obtain local image patches; Upsampling of local image patches; The upsampled local image patch is input into the trained segmentation model to obtain a high-resolution mask; The high-resolution mask is mapped back to the coordinate system of the original video image to generate pixel-level segmentation results.

[0043] like Figure 2 As shown in the lower half, in one embodiment of this application, the model inference process is as follows: 1c, Receive user prompts: User input interaction prompts, including: points (Positive / Negative), bounding boxes, and mask priors. 2c, Initial segmentation prediction: Input the original large image and prompts into the model to generate coarse segmentation results and preliminary target regions; 3c, Adaptive RoI extraction: Using the target center as the anchor point, dynamically crop out local image patches (e.g., with a resolution of 512×512) based on the predicted target range. 4c, Optimal upsampling ratio: Estimate the optimal upsampling ratio based on the target pixel area (suggested range is 1.5–3.0), and upscale to a resolution of 1024×1024 using bicubic interpolation; 5c, Local segmentation refinement: The resampled local image is fed into the segmentation model for fine segmentation to obtain a high-resolution mask; 6c, Coordinate Mapping Backwards: Through the inverse affine transformation matrix, the local mask is accurately mapped back to the original large image coordinate system; 7c, Output the final result: Generate the final pixel-level segmentation result and return it to the user.

[0044] During the inference phase, intelligent cropping and upsampling of RoIs based on prompts are used to map extremely small targets (such as vehicles and small ships) back to the high-resolution feature space, completely avoiding the loss of information under standard input size through downsampling.

[0045] In one embodiment of this application, for drone terminals or vehicle-mounted processing platforms with extremely high real-time requirements, a simplified architecture configuration can be adopted, retaining only the ST-LoRA adapter and Ada-Inference inference pipeline, removing the DHF-Decoder module, and directly using the native SAM decoder. This variant controls the number of new parameters added to the model to within 30% of the original model, and the inference latency is reduced by about 40% compared to the full version. Although the edge detail accuracy is slightly inferior to the full version model, thanks to the adaptive upsampling technology, its recall rate for small targets is still significantly higher than that of the native SAM model.

[0046] Based on SAM, this application addresses the challenges of capturing small targets, temporal discontinuities, and severe background interference in remote sensing scenarios through the following three core technological innovations: 1d, the parameter-efficient spatiotemporal domain adapter ST-LoRA: The LoRA branch is injected in parallel into the self-attention projection layer of the ViT encoder, and a temporal memory link is innovatively introduced. By fine-tuning the last layer block and associating it with the features of the preceding frame, the model not only has the ability to discriminate remote sensing domain features, but also can effectively handle the problems of target occlusion and instantaneous disappearance in video. 2d, High-quality decoder DHF-Decoder with decoupled high-frequency compensation: Inheriting the architectural advantages of High Quality (HQ)-SAM, it further introduces a high-frequency edge refinement branch. This branch integrates early texture features with later semantic context and adopts an alternating optimization strategy with the original decoder to specifically perform geometric constraints on the blurred boundaries of remote sensing targets, significantly improving boundary integrity.

[0047] 3D Adaptive Object-Aware Inference Pipeline Ada-Inference: Reconstructing the Data Processing Chain. During the training phase, dynamic scale jitter and multi-angle rotation enhancement are implemented; during the inference phase, fixed-rate upsampling is abandoned, and a cue-based adaptive RoI extraction technique is adopted. This technique can dynamically adjust the cropping window and perform optimal resampling based on preliminary predictions, greatly enhancing pixel-level resolution for extremely small targets.

[0048] RSVOS-SAM++ demonstrates outstanding performance on authoritative remote sensing video datasets such as SAT-MTB. Compared to the basic SAM dataset, it improves the Intersection over Union (IoU) by over 10% and the Boundary Intersection over Union (BIoU) by a significant margin. While maintaining zero-shot generalization ability, it solves the segmentation challenge of "small, dark, chaotic, and fast" targets in remote sensing videos, making it highly valuable for engineering applications.

[0049] The interactive segmentation apparatus for moving targets in remote sensing video provided in this application will be described below. The interactive segmentation apparatus for moving targets in remote sensing video described below can be referred to in correspondence with the interactive segmentation method for moving targets in remote sensing video described above.

[0050] Figure 5 This is a schematic diagram of the structure of an interactive segmentation device for moving targets in remote sensing video provided in an embodiment of this application, as shown below. Figure 5 As shown, the device 500 includes: Training module 510 is used to train the segmentation model using a pre-acquired dataset to obtain a trained segmentation model, which includes a spatiotemporal enhancement image encoder, a cue encoder, a mask decoder, and a decoupled high-frequency compensation decoder. The segmentation module 520 is used to input the remote sensing image or video to be segmented into the trained segmentation model to obtain the segmentation result; The spatiotemporal augmented image encoder includes a low-rank adaptation matrix and a temporal memory link for extracting features from the input video image; A prompt encoder is used to transform user-input prompts into high-dimensional prompt vectors; The mask decoder is used to obtain preliminary segmentation results based on the features of the input video image and the high-dimensional cue vector; The decoupled high-frequency compensation decoder is used to extract shallow high-frequency features from the features extracted by the spatiotemporally enhanced image encoder, and to optimize the preliminary segmentation results based on the shallow high-frequency features.

[0051] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0052] Based on the methods in the above embodiments, Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown in the illustration, this application provides an electronic device that may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions stored in the memory 630 to execute the interactive segmentation method for moving targets in remotely sensed video described in the above embodiment.

[0053] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the interactive segmentation method for moving targets in remotely sensed video described in the various embodiments of this application.

[0054] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the interactive segmentation method for moving targets in remote sensing video as described in the above embodiments.

[0055] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the interactive segmentation method for moving targets in remote sensing video as described in the above embodiments.

[0056] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0057] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0058] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0059] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0060] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An interactive segmentation method for moving targets in remotely sensed video, characterized in that, include: The segmentation model is trained using a pre-acquired dataset to obtain a trained segmentation model, which includes a spatiotemporal enhancement image encoder, a cue encoder, a mask decoder, and a decoupled high-frequency compensation decoder. Input the remote sensing image or video to be segmented into the trained segmentation model to obtain the segmentation result; The spatiotemporal enhanced image encoder includes a low-rank adaptation matrix and a temporal memory link, used to extract features from the input video image; The prompt encoder is used to convert user-input interactive prompts into high-dimensional prompt vectors; The mask decoder is used to obtain preliminary segmentation results based on the features of the input video image and the high-dimensional cue vector; The decoupled high-frequency compensation decoder is used to extract shallow high-frequency features from the features extracted by the spatiotemporal enhancement image encoder, and to optimize the preliminary segmentation results based on the shallow high-frequency features.

2. The interactive segmentation method for moving targets in remote sensing video according to claim 1, characterized in that, The training process of the spatiotemporal augmentation image encoder includes: The pre-trained ViT is used as an image encoder, pre-trained weights are loaded, and the backbone parameters are frozen. A low-rank adaptation matrix is ​​injected into the self-attention projection layer of ViT, and the low-rank adaptation matrix is ​​used to capture the texture structure and scale changes of the image. A temporal memory link is constructed to process the encoded features of the (t-1)th frame of the video, generate a temporal bias term, and inject the temporal bias term into the self-attention calculation of the tth frame of the video, where t is a positive integer greater than 1 and less than or equal to the total frame length of the video. Based on the pre-acquired dataset, the low-rank adaptation matrix and the temporal memory link are trained; Unfreeze and fine-tune the last layer of the ViT Transformer Block, injecting remote sensing-specific high-level semantic knowledge.

3. The interactive segmentation method for moving targets in remote sensing video according to claim 1, characterized in that, The training process for the mask decoder and the decoupled high-frequency compensation decoder includes: Intermediate feature maps are extracted from the shallow layer of the spatiotemporal enhanced image encoder; Channel dimensionality reduction is performed on the intermediate feature map so that the dimensionality-reduced intermediate feature map is aligned with the upsampled features of the mask decoder in the channel dimension; The reduced intermediate feature map is fused with the semantic features of the mask decoder to form a decoupled high-frequency compensation path; During training, the parameters of the mask decoder and the parameters of the decoupled high-frequency compensation decoder are updated alternately.

4. The interactive segmentation method for moving targets in remote sensing video according to claim 1, characterized in that, The reasoning process of the trained segmentation model includes: Based on the user-input target area-related prompts and video images, determine the center point of the target area in the video image; The video image is cropped with the center point of the target area as the center to obtain a local image block; Upsample the local image patch; The upsampled local image patch is input into the trained segmentation model to obtain a high-resolution mask; The high-resolution mask is mapped back to the coordinate system of the original video image to generate pixel-level segmentation results.

5. An interactive segmentation device for moving targets in remotely sensed video, characterized in that, include: The training module is used to train the segmentation model using a pre-acquired dataset to obtain a trained segmentation model, which includes a spatiotemporal enhancement image encoder, a cue encoder, a mask decoder, and a decoupled high-frequency compensation decoder. The segmentation module is used to input the remote sensing image or video to be segmented into the trained segmentation model and obtain the segmentation result. The spatiotemporal enhanced image encoder includes a low-rank adaptation matrix and a temporal memory link, used to extract features from the input video image; The prompt encoder is used to convert user-input interactive prompts into high-dimensional prompt vectors; The mask decoder is used to obtain preliminary segmentation results based on the features of the input video image and the high-dimensional cue vector; The decoupled high-frequency compensation decoder is used to extract shallow high-frequency features from the features extracted by the spatiotemporal enhancement image encoder, and to optimize the preliminary segmentation results based on the shallow high-frequency features.

6. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform an interactive segmentation method for a moving target in remotely sensed video as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on a processor, the processor performs the interactive segmentation method for moving targets in remotely sensed video as described in any one of claims 1-4.

8. A computer program product, characterized in that, When the computer program product is run on a processor, the processor performs the interactive segmentation method for moving targets in remotely sensed video as described in any one of claims 1-4.

Images