Remote sensing image instance segmentation method and system based on visual basic model
By combining content-aware and deformable convolutional cue generation methods, the problem of complex target edge contours in remote sensing image instance segmentation is solved, achieving more accurate target segmentation and recognition, generating target instance masks and accurately locating the semantic category of the target.
Patent Information
- Application Number
- CN202410953078.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-22
AI Technical Summary
Existing visual models struggle to effectively handle complex target edge contours in remote sensing image instance segmentation tasks, resulting in insufficient segmentation accuracy.
A cue generation method combining content-aware and deformable convolution is adopted. By learning to generate cue information that conforms to the characteristics of remote sensing image targets, cue information is generated to replace manually created cue information such as points and boxes as input to the SAM mask decoder. Instance segmentation is performed using the SAM image encoder, feature aggregation module, cue generator and SAM image decoder.
It achieves more accurate target segmentation and recognition, can generate target instance masks and accurately locate the semantic category of the target, thus improving the precision and accuracy of remote sensing image segmentation.
Smart Images

Figure CN119006809A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing image instance segmentation method and system based on a visual basic model. Background Art
[0002] Visual Foundation Models (VFMs), such as the Segment Everything Model (SAM), have demonstrated excellent generalization performance in a variety of visual scenarios, injecting new vitality into the field of intelligent image analysis. However, given the uniqueness of remote sensing images, it is often difficult to achieve ideal results by directly applying such models to remote sensing scenes. At the same time, there are also complex target edge contour problems in remote sensing image instance segmentation tasks, such as Figure 1 As shown, the difficulty of accurately segmenting the target is further increased. Summary of the invention
[0003] In view of the above problems, the present invention proposes a remote sensing image instance segmentation method and system based on a visual basic model, and specifically proposes a prompt generation method combining content perception and deformable convolution, which generates prompt information that conforms to the characteristics of remote sensing image targets through learning, replacing manually made prompts such as points and boxes as inputs of the SAM mask decoder. This automatically generated prompt information enables SAM to not only generate target instance masks, but also infer the semantic category of the target and accurately locate the target, thereby achieving more accurate target segmentation and recognition.
[0004] In order to achieve the above object, the present invention adopts the following technical solutions:
[0005] On the one hand, the present invention provides a remote sensing image instance segmentation method based on a visual basic model, comprising:
[0006] Step 1: construct a remote sensing image instance segmentation model based on a visual base model; the remote sensing image instance segmentation model includes a SAM image encoder, a feature aggregation module, a prompt generator and a SAM image decoder; the feature aggregation module is used to aggregate the intermediate layer features output by the SAM image encoder, and the prompt generator includes a content-aware feature recombination module and a deformable convolution module;
[0007] Step 2: Perform instance segmentation on the remote sensing image based on the trained remote sensing image instance segmentation model.
[0008] Furthermore, the remote sensing image instance segmentation model adopts a pre-trained SAM image encoder.
[0009] Furthermore, the feature aggregation module is also used to process the aggregated intermediate layer features through a fused convolutional layer, and the fused convolutional layer is composed of two 3×3 convolutional layers and one 1×1 convolutional layer.
[0010] Furthermore, in the hint generator, the feature map processed by the feature aggregation module is upsampled through the content-aware feature reconstruction module, and the convolution kernel sampling position is adaptively adjusted by using deformable convolution to capture the target contour information. Then, the region proposal network RPN is used to generate candidate target boxes, and the visual feature representation of a single object is extracted from the position-encoded feature map through the RoI pooling layer.
[0011] Furthermore, in the hint generator, after extracting the visual feature representation of a single object, it also includes: inputting the extracted visual feature representation of the single object into a semantic head, a positioning head and a hint head, identifying the target category through the semantic head, establishing a matching criterion between the hint and the target mask through the positioning head, and generating the embedded hint required by the SAM mask decoder through the hint head.
[0012] Furthermore, the deformable convolution module includes deformable convolution DCNv3, layer normalization, feedforward network and GELU activation function.
[0013] Furthermore, the segmentation loss of the SAM mask decoder adopts a combined loss function of Dice loss and cross entropy loss.
[0014] Furthermore, the positioning loss of the RPN adopts a CIoU loss function.
[0015] Another aspect of the present invention provides a remote sensing image instance segmentation system based on a visual basic model, comprising:
[0016] An instance segmentation model construction unit is used to construct a remote sensing image instance segmentation model based on a visual base model; the remote sensing image instance segmentation model includes a SAM image encoder, a feature aggregation module, a prompt generator and a SAM image decoder; the feature aggregation module is used to aggregate the intermediate layer features output by the SAM image encoder, and the prompt generator includes a content-aware feature recombination module and a deformable convolution module;
[0017] The instance segmentation unit is used to perform instance segmentation on the remote sensing image based on the trained remote sensing image instance segmentation model.
[0018] Compared with the prior art, the present invention has the following beneficial effects:
[0019] The present invention proposes a remote sensing image instance segmentation method and system based on a visual basic model, and specifically proposes a prompt generation method combining content perception and deformable convolution, which generates prompt information that conforms to the characteristics of remote sensing image targets through learning, replacing manually produced prompts such as points and boxes as inputs of the SAM mask decoder. This automatically generated prompt information enables SAM to not only generate target instance masks, but also infer the semantic category of the target and accurately locate the target, thereby achieving more accurate target segmentation and recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 It is an example of the problems existing in the remote sensing image instance segmentation task in the background technology;
[0021] Figure 2 A flowchart of a remote sensing image instance segmentation method based on a visual basic model according to an embodiment of the present invention;
[0022] Figure 3 A schematic diagram of a framework of a base remote sensing image instance segmentation model constructed according to an embodiment of the present invention;
[0023] Figure 4 A schematic diagram of the structure of a feature aggregation module constructed for an embodiment of the present invention;
[0024] Figure 5 A schematic diagram of the structure of a prompt generator constructed for an embodiment of the present invention;
[0025] Figure 6 A comparison of the different sampling methods provided by the embodiments of the present invention;
[0026] Figure 7 A schematic diagram of the structure of a deformable convolution module constructed for an embodiment of the present invention;
[0027] Figure 8 The present invention is a schematic diagram of the architecture of a remote sensing image instance segmentation system based on a visual basic model according to an embodiment of the present invention. DETAILED DESCRIPTION
[0028] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:
[0029] like Figure 2 As shown, a remote sensing image instance segmentation method based on a visual basic model includes:
[0030] S101: constructing a remote sensing image instance segmentation model based on a visual base model; the remote sensing image instance segmentation model includes a SAM image encoder, a feature aggregation module, a prompt generator and a SAM image decoder; the feature aggregation module is used to aggregate the intermediate layer features output by the SAM image encoder, and the prompt generator includes a content-aware feature recombination module and a deformable convolution module;
[0031] S102: Perform instance segmentation on the remote sensing image based on the trained remote sensing image instance segmentation model.
[0032] Furthermore, the overall framework of the remote sensing image instance segmentation model constructed by the method of the present invention is as follows: Figure 3 As shown in Figure 1, it includes four parts: SAM image encoder, feature aggregation module, prompt generator and SAM mask decoder. First, the image is processed by the pre-trained SAM image encoder to generate the key feature map F. img and multiple intermediate feature maps F i Among them, F img Interacts with the SAM mask decoder and the generated prompt to obtain a segmentation mask guided by the prompt. i It is used for feature aggregation and prompt generation.
[0033] In order to improve the quality of features, the content-aware feature recombination module (CARAFE) is used in the prompt generator to aggregate the feature map F agg Upsampling is performed to obtain feature maps with richer details and higher reconstruction quality. At the same time, the deformable convolution module (DCNv3) is introduced to improve the model's ability to capture target contour information by using its ability to adaptively adjust the convolution kernel sampling position.
[0034] Next, RPN (Region Proposal Network) is used to perform target recall in multi-scale features and generate candidate target frames. Through the RoI pooling layer, the present invention extracts the visual feature representation of a single object from the feature map after position encoding. These visual features further derive semantic heads, positioning heads, and prompt heads. The semantic head is responsible for determining the specific category of the target, and the positioning head establishes a greedy matching criterion based on positioning for matching the generated prompts and target instance masks. The prompt head is responsible for generating the prompt embedding required by the SAM mask decoder.
[0035] Finally, F imgThe generated prompt interacts in the SAM mask decoder to generate the target instance mask by cross-attention. This method performs well in preserving details and reconstruction quality, and can generate semantically recognizable segmentation results.
[0036] The whole process of the method of the present invention can be described as follows: For the training set D train ={(x1,y1),…,(x N ,y N )}, where x i ∈R H×W×3 Represents a remote sensing image, y i = {b i ,c i ,m i} represents the remote sensing image x i The corresponding label information includes x i The bounding boxes of the n objects contained in i ∈R n×4 ), the corresponding semantic category Binary Mask Through these label information, a prompt information suitable for SAM is trained, so that SAM can better understand and process images. In the test phase, SAM can locate the target of interest in the image, accurately identify the target category, and accurately segment the target mask based on the prompt information. The process can be described as follows:
[0037] F img ,{F i}=Φ i-enc (x k ) (1)
[0038] F agg =Φ aggregator ({F i}) (2)
[0039] {T j ,c j}=Φ prompter (F agg ) (3)
[0040] Μ j =Φ m-dec (F img ,[T out ,T j ]) (4)
[0041] The image is passed through a SAM image encoder with frozen parameters (Φ i-enc ) is processed to generate feature-rich F img ∈R h×w×cand multiple intermediate feature maps F i , where {F i}∈R k×h×w×c , i∈{1,…,k}, represents the multi-layer semantic feature map output by the image encoder. Next, F i Through the feature aggregation module (Φ aggregator ) and prompt generator (Φ prompter ) after processing, multiple groups are obtained and its corresponding semantic category (c j ∈R C ,j∈{1,…,N p}). Among them, k p Indicates the number of prompt embeddings required for each mask generation. j Contains only prompts for foreground target instances, whose semantic category is c j Each T j are combinations of multiple prompts, which can be in the form of point embeddings or boxes representing the boundaries of the target, so as to more accurately describe the shape and position of the target instance. Finally, after the SAM mask decoder (Φ m-dec ) is promoted by the cross-attention mechanism in F img With the generated prompt information T j And pre-inserted learnable prompt information T out After interaction, the target mask M is generated j . T out ∈R 5×c Represents the pre-inserted learnable tokens, representing four different masks and their corresponding IoU predictions.
[0042] 1SAM Image Encoder
[0043] The SAM image encoder, or Vision Transformer, is pre-trained on a large dataset of more than one billion instances. Since it has shown excellent performance in segmentation tasks and has a large number of model parameters, it may be challenging to adjust it when the amount of experimental data is relatively small. Therefore, during the model training process, the strategy of freezing the image encoder parameters is adopted, that is, directly using its pre-trained parameters without further fine-tuning. The advantage of this is that the feature extraction capability of the pre-trained model can be fully utilized while avoiding the overfitting problem caused by insufficient data.
[0044] 2 Feature Aggregation Module
[0045] Since SAM is a category-independent segmentation model, in order to obtain semantically relevant and discriminative features, a Figure 4 The feature aggregation module shown aggregates the intermediate layer features F output by the image encoder i , then F i After fusing the convolutional layers, we get F agg .
[0046] This module can be described recursively as follows:
[0047]
[0048] In the above formula, F i ∈R h×w×c and They represent the intermediate layer features output by the SAM image encoder and the features obtained by Φ DownConv Process the downsampled features. This process first uses a 1×1 convolution block (convolution, normalization, ReLU activation function) to reduce the number of channels from c to Then use a 3×3 convolution block with a stride of 2 to reduce the spatial dimension. Φ Conv2D represents a 3×3 convolutional ReLU block, and Φ FusionConv Represents the final fused convolutional layer, which consists of two 3×3 convolutional layers and one 1×1 convolutional layer, used to restore the original channel size of the SAM mask decoder.
[0049] 3. Hint Generator Combining Content-Aware and Deformable Convolution
[0050] After fusing the semantic features, the hint generator is used to generate embedding hints for the SAM mask decoder. The hint generator structure is as follows: Figure 5 shown.
[0051] After feature aggregation, single-scale features are obtained, while multi-scale features are crucial for target detection and instance segmentation tasks. At the same time, in order to ensure that it matches the input of the SAM encoder, the image is scaled to 1024×1024 pixels in the input stage, resulting in varying degrees of deformation of the target in the image. Therefore, the method of the present invention is based on this single-scale feature, and by combining a content-aware module and a deformable convolution module, a corrected multi-scale feature is constructed. This method can improve the model's ability to perceive changes in target scale and deformation, so that it can more accurately locate the target and generate more appropriate prompt information.
[0052] (1) Content-aware feature reconstruction module
[0053] For tasks such as segmentation that require fine upsampling, detail preservation and reconstruction quality during the upsampling process are crucial to improving the overall performance of the model.
[0054] Common upsampling methods include nearest neighbor, bilinear, and transposed convolution (Deconvolution), and different upsampling methods have different characteristics. The nearest neighbor or bilinear upsampling method is mainly based on the spatial position of the pixel point, with small amount of calculation, but does not consider the semantic information of the feature map, and the receptive field is also small (the receptive field of the nearest neighbor is 1×1, and the receptive field of the bilinear is 2×2). Although the transposed convolution performs upsampling by learning the upsampling kernel, the same kernel is applied to each position of the feature map, which cannot fully capture the content information of the feature map. In addition, when the kernel size is large, the transposed convolution will introduce a large number of parameters and computational complexity.
[0055] Different from the above two methods, the Content-Aware ReAssembly of Features (CARAFE) uses an adaptive method to accurately upsample according to the image content to improve the resolution and quality of the convolutional neural network feature map. This method has instance-specific content perception and can dynamically generate kernels that conform to the current feature map content. At the same time, CARAFE can gather and integrate contextual information within a larger receptive field to ensure that the upsampling process is more in line with the actual image content, thereby improving the reconstruction accuracy of the features. Its core idea is to use the underlying content information to dynamically predict and generate recombinant kernels, which are then used to recombine features in a predefined neighboring area to form a more refined and accurate feature map.
[0056] Specifically, CARAFE consists of two main steps. First, it generates a unique recombinant kernel for each target position based on the content of the input features through a kernel prediction module. Given a feature map X of size H×W×C and an upsampling ratio U, CARAFE can efficiently generate a new high-resolution feature map X' of size UH×UW×C. For any target position l'=(i',j') on the output feature map X', there is a corresponding source position l=(i,j) at the input X, where i=[i' / σ], j=[j' / σ]. Where N(X l ,k) is represented as a k×k subregion of X centered at position l, that is, X l These location-specific kernels are then used in a content-aware recombination module to accurately recombine features.
[0057] In the first step, the kernel prediction module Ψ is based on X l Neighborhood, predict a position-dependent kernel W for each position l' l' , as shown in Equation (6). The reorganization step is given by Equation (7), where Φ is the content-aware reorganization module, which uses the kernel W l' Reassemble X l 's neighborhood.
[0058] W l' =Ψ(N(X l ,k encoder )) (6)
[0059] X l ”=φ(N(X l ,k up ),W l' ) (7)
[0060] Among them, k encoder Indicates a kernel size of k encoder The convolutional layer, k up Indicates that the kernel size required for each target position is k up of the recombinant nucleus.
[0061] (2) Deformable Convolution Block (DCN Block)
[0062] As an extension of the DCN sequence, DCNv3 introduces a multi-grouping mechanism based on DCNv2, realizes the sharing of weighted convolution neurons, and normalizes the modulation scalars between sampling points, while maintaining high efficiency and improving the accuracy and flexibility of the model. DCNv3 has the following two advantages:
[0063] First, DCNv3 effectively makes up for the shortcomings of conventional convolution in long-range dependency and adaptive spatial aggregation. Through dynamic learning and sparse sampling, it can better capture contextual information in images and improve the accuracy and flexibility of the model.
[0064] Secondly, compared with the multi-head self-attention (MHSA) and deformable multi-head attention used in Transformer, DCNv3 also has the inductive bias capability of convolution, which can make the model more efficient with less training data and shorter training time.
[0065] The expression of DCNv3 is as follows:
[0066]
[0067] Where G is the total number of aggregation groups. For the gth group, w g ∈R C×C ' represents the position-independent projection weight of the group, where C' = C / G represents the group dimension. gk ∈R represents the modulation scalar of the kth sampling point in the gth group, normalized by the softmax function along the dimension. g ∈R C ' ×H×W Represents the slice input feature map. △p gkis the grid sampling position p in group g k The corresponding offset.
[0068] Figure 6 The spatial information aggregation capabilities of several methods are compared in Figure 6 Although the Multi-Head Self-Attention (MHSA) in (a) has the ability to aggregate global information, its computational and memory costs are high when processing downstream tasks that require high-resolution input. Figure 6 The action range of MHSA in (b) is limited to a local window, which reduces the computational cost while maintaining a certain performance. Figure 6 (c) shows a deep convolution using a large convolution kernel. Although it can model long-distance dependencies and is comparable to MHSA to a certain extent, it lacks the necessary adaptive ability in spatial aggregation. Figure 6 The deformable convolution in (d) combines the advantages of MHSA with the characteristics of sparse sampling, maintaining high performance while having a smaller amount of computation.
[0069] Thanks to the above advantages, when using DCNv3 for remote sensing image target feature extraction, the model can adaptively adjust the sampling offset and modulation scalar according to the target characteristics, thereby focusing on the target itself, reducing the interference of complex background information, and effectively dealing with problems such as target scale change, rotation and deformation in remote sensing images.
[0070] In order to better utilize the performance of DCNv3, DCNv3 is combined with the components used in Transformer, including layer normalization (LN), feedforward network (FFN), and GELU activation function, such as Figure 7 This combination strategy enables the model to better capture subtle changes in the target.
[0071] Next, the multi-scale features are passed through an anchor-based region proposal network (RPN) to generate candidate target boxes, and then the RoI pooling layer is used to extract the visual features of a single target from the position encoding feature map. These features are input into three key parts, namely the semantic head, the localization head, and the prompt head. Among them, the semantic head is responsible for identifying the target category, and the localization head is responsible for establishing the matching criteria between the prompt and the target mask (greedy matching based on IoU). Finally, the prompt head generates the embedded prompts required by the SAM mask decoder. This process can be expressed using the following formula:
[0072]
[0073] Among them, Φ rpn represents a lightweight region proposal network (RPN). In order to prevent Φ roi-pThe operation causes the subsequent prompt generation to lose information related to the entire image position, merging the position encoding (PE) into the original fusion feature (F agg ). Function Φ cls , Φ reg and Φ prompt They correspond to the semantic head, positioning head and prompt head respectively. j Contains only hints for foreground object instances, a single T j It is a combination of multiple prompts, i.e., it represents an instance mask with multiple point embeddings or bounding boxes. Considering that it is difficult for neural networks to accurately predict high-frequency information, a sinusoidal function is used to generate high-frequency information to align the embedding in the prompt encoder of SAM and the generated prompt embedding features.
[0074] 4 Loss Function
[0075] In the method of the present invention, multiple loss functions are involved, including the binary classification loss and positioning loss of the RPN network, the classification loss of the semantic head, the regression loss of the positioning head, and the segmentation loss of the SAM mask decoder. In the basic version of the method, the cross entropy loss function is used for both classification and segmentation tasks, while the Smooth L1 loss function is used for the positioning task.
[0076] (1) Segmentation subtask loss function
[0077] In the segmentation subtask, the cross entropy loss function is mainly used for pixel classification, that is, the accuracy of dividing pixels into foreground (target of interest) and background. This is somewhat limited when dealing with accurate segmentation of target boundaries in complex backgrounds in remote sensing images. In contrast, the Dice loss function pays more attention to the overlap between the predicted segmented pixels and the ground truth mask, which helps to make the predicted foreground area closer to the true value. Therefore, in order to balance the performance of the model at the pixel level and the category level, and avoid excessive focus on one aspect and resulting in performance degradation, the method of the present invention introduces the Dice loss function and the cross entropy loss function for joint training to improve the model segmentation performance. The Dice loss function formula is as follows:
[0078]
[0079] In the above formula, X and Y represent the predicted and true foreground pixel sets, respectively. As can be seen from the above formula, Dice loss is actually a generalization of IoU loss in the segmentation task, the difference is that the segmentation task involves predicting irregular polygonal areas. In the segmentation subtask, the method of the present invention adopts a combined loss function of Dice loss and cross entropy loss, and the calculation formula is as follows:
[0080] L Seg =L Dice +L CE (11)
[0081]
[0082] In the above formula, M represents the number of categories, y ic is a sign function that takes values of 0 or 1. This means that if the true category of sample i is c, then y ic is equal to 1, otherwise it is equal to 0. ic represents the predicted probability that sample i belongs to category c.
[0083] (2) Localization subtask loss function
[0084] In the positioning subtask, the positioning loss of the baseline method uses the Smooth L1 loss function, but the SmoothL1 loss function only optimizes the predicted coordinate offset and cannot directly reflect the regression quality of the bounding box. In order to solve this problem, the method of the present invention introduces the CIoU loss function, which combines the concepts of IoU and GIOU and can more accurately measure the similarity between the predicted box and the true box. At the same time, the CIoU loss function also alleviates the gradient invariance problem in non-overlapping cases, which can further improve the positioning accuracy of the model. The positioning loss function used by the method of the present invention is expressed as follows:
[0085]
[0086] Among them, b represents the center point of the predicted bounding box, b gt Represents the center point of the ground-truth box. 2 represents the square of the distance between the two center points. c represents the distance between b and b gt The minimum enclosing rectangle diagonal length of two rectangles. α is a balance parameter, while v is a parameter that measures the consistency of the aspect ratio. The calculation formulas for α and v are as follows:
[0087]
[0088] Among them, w and h represent the height and width of the prediction box respectively, and w gt 、h gt denote the height and width of the ground truth box, respectively.
[0089] Therefore, the overall loss can be expressed as:
[0090]
[0091] Among them, L cls represents the cross entropy loss calculated between the predicted category and the target, L reg represents the CIOU loss for calculating the target offset between the predicted box and the true box. In addition, L segrepresents the cross entropy and Dice joint loss between the SAM decoded mask and the target instance mask label, where the matching criterion is determined by the IoU of the bounding box and the indicator function I confirms the positive sample match. Finally, L rpn Represents the loss of region proposal.
[0092] Based on the above embodiments, Figure 8 As shown, the present invention also proposes a remote sensing image instance segmentation system based on a visual basic model, comprising:
[0093] An instance segmentation model construction unit is used to construct a remote sensing image instance segmentation model based on a visual base model; the remote sensing image instance segmentation model includes a SAM image encoder, a feature aggregation module, a prompt generator and a SAM image decoder; the feature aggregation module is used to aggregate the intermediate layer features output by the SAM image encoder, and the prompt generator includes a content-aware feature recombination module and a deformable convolution module;
[0094] The instance segmentation unit is used to perform instance segmentation on the remote sensing image based on the trained remote sensing image instance segmentation model.
[0095] In summary, the present invention generates prompt information that conforms to the characteristics of remote sensing image targets through learning, replacing manually produced prompts such as points and boxes as the input of the SAM mask decoder. This automatically generated prompt information enables SAM to not only generate target instance masks, but also infer the semantic category of the target and accurately locate the target, thereby achieving more accurate target segmentation and recognition.
[0096] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.
Claims
1. A remote sensing image instance segmentation method based on a visual basic model, It is characterized in that include: Step 1: construct a remote sensing image instance segmentation model based on a visual base model; the remote sensing image instance segmentation model includes a SAM image encoder, a feature aggregation module, a prompt generator and a SAM image decoder; the feature aggregation module is used to aggregate the intermediate layer features output by the SAM image encoder, and the prompt generator includes a content-aware feature recombination module and a deformable convolution module; Step 2: Perform instance segmentation on the remote sensing image based on the trained remote sensing image instance segmentation model.
2. The remote sensing image instance segmentation method based on a visual base model according to claim 1, It is characterized in that The remote sensing image instance segmentation model adopts a pre-trained SAM image encoder.
3. The remote sensing image instance segmentation method based on a visual base model according to claim 1, It is characterized in that The feature aggregation module is also used to process the aggregated intermediate layer features through a fused convolutional layer, where the fused convolutional layer consists of two 3×3 convolutional layers and one 1×1 convolutional layer.
4. The remote sensing image instance segmentation method based on a visual base model according to claim 1, It is characterized in that In the hint generator, the feature map processed by the feature aggregation module is upsampled by the content-aware feature reconstruction module, and the convolution kernel sampling position is adaptively adjusted by deformable convolution to capture the target contour information. Then, the region proposal network RPN is used to generate candidate target boxes, and the visual feature representation of a single object is extracted from the position-encoded feature map by the RoI pooling layer.
5. The remote sensing image instance segmentation method based on a visual base model according to claim 4, It is characterized in that In the prompt generator, after extracting the visual feature representation of a single object, it also includes: inputting the extracted visual feature representation of the single object into a semantic head, a positioning head and a prompt head, identifying the target category through the semantic head, establishing a matching criterion between the prompt and the target mask through the positioning head, and generating the embedded prompt required by the SAM mask decoder through the prompt head.
6. The remote sensing image instance segmentation method based on a visual base model according to claim 1, It is characterized in that The deformable convolution module includes deformable convolution DCNv3, layer normalization, feedforward network and GELU activation function.
7. The remote sensing image instance segmentation method based on a visual base model according to claim 1, It is characterized in that The segmentation loss of the SAM mask decoder adopts a combined loss function of Dice loss and cross entropy loss.
8. The remote sensing image instance segmentation method based on a visual base model according to claim 4, It is characterized in that The positioning loss of the RPN adopts the CIoU loss function.
9. A remote sensing image instance segmentation system based on a visual basic model. It is characterized in that include: An instance segmentation model construction unit is used to construct a remote sensing image instance segmentation model based on a visual base model; the remote sensing image instance segmentation model includes a SAM image encoder, a feature aggregation module, a prompt generator and a SAM image decoder; the feature aggregation module is used to aggregate the intermediate layer features output by the SAM image encoder, and the prompt generator includes a content-aware feature recombination module and a deformable convolution module; The instance segmentation unit is used to perform instance segmentation on the remote sensing image based on the trained remote sensing image instance segmentation model.
Citation Information
Patent Citations
Remote sensing image lightweight semantic segmentation method based on edge decoupling
CN113159051A
High-resolution remote sensing image classification method and system based on deep learning
CN114694038A
Remote sensing image target detection method
CN116258953A
Remote sensing image interactive segmentation method based on visual basis large model fine tuning
CN117197461A
Remote sensing image instance segmentation method and model based on dynamic convolution enhancement
CN117877034A
Cited By
Image processing method and device, equipment and storage medium
CN119380034A
Image processing method, device, equipment and storage medium
CN119380034B
Remote sensing image geographic element automatic identification method and system based on large model analysis
CN119445582A
Automatic Recognition Method and System for Geographical Elements of Remote Sensing Images Based on Large Model Analysis
CN119445582B
Mask prompt-based remote sensing ground feature regular vector contour extraction method and device
CN120374658A
