Image segmentation method for camouflaged target detection based on DINOv3 and SAM2

Abstract

This invention belongs to the field of computer vision technology, specifically relating to an image segmentation method for camouflaged target detection based on DINOv3 and SAM2. This method constructs self-supervised visual and image structure branches in parallel to extract semantic and geometric features. An automatic prompt generation module receives semantic features and generates a pixel-level foreground probability map, then autonomously extracts point, bounding box, and mask prompts. A deformable spatial channel attention fusion module is used to achieve fine-grained alignment of cross-model features through deformable convolution and channel weighting. A mask decoding module combines prompt embedding and fused features to output a segmentation mask. This invention solves the problems of blurred boundaries of camouflaged targets and dependence on prompts, improving detection accuracy in low-contrast scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to an image segmentation method for camouflaged target detection based on DINOv3 and SAM2. Background Technology

[0002] Camouflaged object detection (COD), a highly challenging frontier topic in computer vision, aims to accurately identify and segment deliberately hidden targets in complex environments where appearance, color, and texture are highly integrated with the background. Compared to traditional salient object detection, camouflaged targets naturally exhibit low contrast, blurred boundaries, and high structural consistency. Their visual cues are highly diffuse in the spatial dimension, making conventional segmentation algorithms based on gradient changes or color abrupt changes often ineffective. With the evolution of deep learning technology, early COD solutions focused on heuristic modeling for specific camouflage patterns. For example, they constructed two-stage networks by simulating the "search-recognition" mechanism in the biological world, or utilized reverse attention mechanisms, multi-scale feature extraction, and long-range dependency modeling to enhance the ability to capture weak targets. Although such specialized models perform well on specific benchmark datasets, they often rely on highly customized network topologies and specific prior assumptions. When facing complex real-world scenarios across different domains, the generalization bottleneck and transfer robustness of these models remain significant.

[0003] In recent years, large-scale vision foundational models, represented by the Segment Anything Model (SAM) and its evolved version SAM2, have provided a new technological paradigm for image segmentation tasks due to their outstanding generalization ability demonstrated under massive pre-training data. SAM2, while inheriting SAM's powerful zero-shot segmentation capabilities, significantly improves the model's adaptability to diverse scenarios by introducing a more robust feature modeling mechanism and interactive decoding strategy. However, when SAM2 is directly applied to camouflaged target detection tasks, its performance exhibits a significant degradation. The underlying reason is that the core architecture of the SAM series models is based on "prompt-driven" principles. Its mask decoder, when processing segmentation tasks, heavily relies on geometric edge signals in the input features and externally provided cues. In typical camouflage scenarios, the semantic boundary between the target and the background is extremely compressed, lacking clear and stable structural contours. This leads to secondary problems in SAM2's image encoder when extracting features, such as overly smooth masks or omissions of core target regions, making it unable to spontaneously form a complete semantic model of the camouflaged target without strong cues.

[0004] Existing SAM-based COD improvement schemes generally face a dual technical contradiction: "reliability of cue information" and "feature semantic mismatch." On the one hand, to compensate for SAM's insufficient localization ability in weak perception environments, existing technologies usually require manually labeled points, bounding boxes, or coarse masks as external cue inputs. However, in real-world applications, camouflaged targets exist to evade visual observation, and even for human observers, obtaining high-quality point or bounding box cues involves high uncertainty and operational costs. This high dependence on human intervention severely restricts the model's practical value in automated processing. On the other hand, SAM2's feature representation essentially focuses more on fine-grained characterization of local textures and boundaries, lacking a macroscopic grasp of global semantic logic. Even with the introduction of self-supervised visual models with strong global semantic discriminative capabilities, such as DINOv3, the inherent differences in their pre-training targets and feature space distributions make it difficult to achieve high-performance automatic camouflaged target detection. This necessitates a deep synergy between the "global semantic consistency" captured by DINOv3 and the "local geometric accuracy" of SAM2, rather than simple feature stacking.

[0005] When the high-level semantic features extracted by DINOv3 are fused with the shallow structural features of SAM2, if there is a lack of effective spatial alignment and channel selection mechanisms, the model is prone to generating false alarms in complex background noise, or feature collapse at semantic boundaries, resulting in the final segmentation mask failing to maintain both target integrity and fine-grained edge perception.

[0006] Therefore, how to construct an automated prompt generation mechanism that does not require human intervention, and design a sophisticated cross-model feature alignment and fusion module to solve the inherent contradiction between the difficulty of semantic localization and the fuzzy boundary recovery in camouflaged scenes, has become a key challenge and technical problem that urgently needs to be solved in the field of computer vision and intelligent image processing. Summary of the Invention

[0007] To achieve the above-mentioned objectives, this invention provides an image segmentation method for camouflaged target detection based on DINOv3 and SAM2. This method solves the technical problems of blurred boundaries, semantic loss, and high dependence on manual prompts caused by the high fusion of target and background in camouflaged target detection by constructing an automatic prompt generation and multi-scale feature fusion architecture guided by a self-supervised visual basic model.

[0008] The camouflaged target detection image segmentation method based on DINOv3 and SAM2 described in this invention is composed of a self-supervised visual feature extraction branch, an image structure feature extraction branch, an automatic prompt generation module, a deformable spatial channel attention fusion module (DSAF module), and a mask decoding module working together.

[0009] To achieve the above-mentioned objectives, this invention provides an image segmentation method for camouflaged target detection based on DINOv3 and SAM2. This method solves the technical problems of blurred boundaries, semantic loss, and high dependence on manual prompts caused by the high fusion of target and background in camouflaged target detection by constructing an automatic prompt generation and multi-scale feature fusion architecture guided by a self-supervised visual basic model.

[0010] The camouflaged target detection image segmentation method based on DINOv3 and SAM2 described in this invention is composed of a self-supervised visual feature extraction branch, an image structure feature extraction branch, an automatic prompt generation module, a deformable spatial channel attention fusion module, and a mask decoding module working together.

[0011] In the self-supervised visual feature extraction branch, this invention employs a self-supervised pre-trained model based on the visual Transformer architecture as the first feature extractor to capture global semantic consistency features in the input image. The first feature extractor extracts features from the input image... Block projection and multi-layer self-attention computation are performed to extract four-level semantic feature maps with different spatial resolutions, denoted as follows: F d1 , F d2 , F d3 , F d4 The corresponding downsampling ratios are 4, 8, 16, and 32, respectively. The first feature extractor utilizes discriminative representations learned on a large-scale unsupervised dataset to maintain a stable response to camouflaged regions in low-contrast scenes, providing basic semantic support for subsequent prompt generation.

[0012] Furthermore, in the image structure feature extraction branch, this invention employs a second feature extractor based on a hierarchical visual encoder to extract the local geometric structure and fine-grained edge information of the input image. The second feature extractor operates in parallel with the first feature extractor, outputting four sets of multi-scale geometric feature maps through layer-by-layer convolution and spatial downsampling operations, denoted as follows: F s1 , F s2 , F s3 , F s4 The second feature extractor focuses on capturing high-frequency components in its feature space, which can effectively compensate for the shortcomings of the first feature extractor in characterizing local topology.

[0013] As a core component of this invention, the automatic suggestion generation module receives multi-scale semantic features output by the first feature extractor. F di The automatic suggestion generation module constructs semantically enhanced suggestion priors through layer-by-layer upsampling and feature alignment mechanisms. Specifically, the automatic suggestion generation module first processes the highest-level features... F d4 After 1×1 convolution compression, the signal is upsampled using bilinear interpolation. F d3 The scale, and with F d3 Channel splicing is performed, followed by fusion using a set of 3×3 residual convolutional layers to obtain fused features. F m3 And so on, through the formula Generate sequentially F m2 and F m1 During the training phase, the fused features F mi Predict the corresponding pixel-level foreground probability map using a lightweight fully convolutional layer. Furthermore, it employs deep supervision via a truth mask to ensure that the automatic prompt generation module can accurately pinpoint the potential areas of the camouflaged target.

[0014] Based on the foreground probability map P i The automatic prompt generation module executes automated prompt extraction logic to eliminate the need for manual intervention. First, the module sets a fixed threshold... τ Binarize P1 to obtain a coarse segmentation mask. M prompt The binarization process is expressed as: if ,but Otherwise, it is 0. M prompt It is directly used as a mask hint and sent to the subsequent encoding stage. Secondly, the module... M prompt Perform a spatial scan of the connected regions in the graph, calculate their minimum bounding rectangle, and generate bounding box hints. ;like M prompt If it is an empty set, then... B prompt Set to a vector of all zeros. Next, the module selects the vector with the highest response value from the probability graph P1. K The pixel coordinates are used as a point hint. The midpoint position is through Argmax The operator is extracted iteratively from the global response graph, andK The value of is fixed at 3. M prompt 、B prompt 、P points Together, they form an automatically generated set of prompts, effectively solving the bottleneck of difficulty in obtaining manual prompts in camouflaged environments.

[0015] Furthermore, to achieve refined alignment of features across models, this invention designs a deformable spatial channel attention fusion module. The fusion module is positioned between corresponding levels of the first and second feature extractors. For the first... i The feature fusion process of the layers first combines semantic features F mi Projecting the geometric features through a 1×1 convolution F si With the same channel dimension, projection features are obtained. F mi ' Subsequently, the projected features are fed into a deformable spatial attention unit, which uses a deformable fringe convolution operator to adaptively offset the spatial sampling position. The deformable fringe convolution decomposes the two-dimensional offset into one-dimensional irregular samples in the horizontal and vertical directions, with an offset Δ... p Through the F mi ' The spatial sampling map is obtained through lightweight convolution computation. During the sampling process, linear interpolation is used to obtain feature values ​​at non-integer coordinates, generating the spatial sampling map. F sample : ,in W n This is a learnable weight matrix.

[0016] Within the deformable spatial channel attention fusion module, a channel adaptive weighting mechanism is further introduced. This module extracts data through a global average pooling layer. F sample The channel statistics are calculated, and the channel attention weights are calculated using two fully connected layers and a Sigmoid activation function. The final generated joint attention map A This is the element-wise product of the spatial sampling weights and the channel weights. The enhanced features after fusion. F fused,i Calculated using the following formula: , where ⊙ denotes element-wise weighting. This design guides the redistribution of geometric features through semantic priors, enabling the model to spontaneously focus on the weak boundary regions of camouflaged targets, achieving a deep coupling between global semantic consistency and local structural accuracy.

[0017] The automatically generated set of prompts in the mask decoding module M prompt 、B prompt 、P points First, the input is fed into the cue encoder, which transforms it into a cue embedding vector in a high-dimensional space. The cue encoder uses sine and cosine position encoding for point coordinates, linear projection encoding for bounding box information, and downsampling convolution encoding for mask information. Subsequently, the cue embedding vector is fused with multi-scale features processed by the deformable space channel attention fusion module. F fused,i The features are fed into a mask decoder. This mask decoder incorporates a multi-layer bidirectional cross-attention mechanism, which dynamically interacts with the cue embeddings to gradually refine the boundary representation of the disguised target. In the final layer of the decoding process, a transposed convolution restores the feature map to the original image resolution, outputting the final segmentation mask. S p .

[0018] As a key aspect of the completeness of the technical solution of this invention, the method described herein employs a multi-task joint loss function for training and optimization to balance pixel-level classification accuracy with the overall consistency of the target structure. The joint loss function... L total It is a weighted sum of the decoder output loss and the auto-completion generation module output loss. Specifically, each loss term is a weighted binary cross-entropy loss. L wbce Weighted intersection and comparison loss L wiou Composition. The weighted binary cross-entropy loss. L wbce The mathematical expression is: ; in, g ij The true pixel value. p ij To predict probability values, w ij The spatial weighting factor is calculated based on the truth mask. It is obtained by performing average pooling on the truth mask and calculating the difference between the central region and the edge region, aiming to increase the loss weight of pixels in the boundary region.

[0019] The weighted crossover ratio lossL wiou The mathematical expression for the group is: ; By introducing spatial weighting factors w ij This loss function significantly enhances the model's ability to capture the subtle structures of camouflaged targets. The total loss function is defined as: ; Where G is the truth mask, S p This is the final output segmentation probability map. S i λ is the intermediate probability plot generated by the auto-completion module. i The preset hyperparameter weights were set to 0.5, 0.3, and 0.2 respectively; weighted binary cross-entropy loss. L wbce Weighted intersection and comparison loss L wiou Spatial weighting factors were introduced in all of them. w ij， Spatial weighting factor w ij This is achieved by performing average pooling on the ground truth mask and calculating the difference between the central and edge regions. The aim is to increase the loss weight of pixels in the boundary region of the camouflaged target, thereby enhancing the model's ability to capture weak boundaries.

[0020] In summary, the camouflage target detection image segmentation method based on DINOv3 and SAM2 described in this invention establishes a dual feature foundation of semantics and structure through a parallel architecture of a first feature extractor and a second feature extractor. The automatic prompt generation module achieves autonomous transformation from semantic representation to spatial prompts, completely eliminating the constraints of manual prompts. The deformable spatial channel attention fusion module completes precise alignment and enhancement of cross-model features, resolving feature mismatch contradictions in camouflage scenes. While maintaining the generalization advantages of large-scale visual basic models, this invention improves detection accuracy and robustness in low-contrast, weak-boundary complex environments through refined engineering design.

[0021] Furthermore, the sampling point offset calculation in the deformable spatial attention unit follows the following physical logic: for each spatial location in the input feature map... p The unit predicts that N Offset Δ of each sampling point p n and the corresponding sampling weights m n The sampling process is represented as follows: ; in, W n This represents the learnable weight matrix. By applying this operator to the edge regions of camouflaged targets, the model can overcome background noise and acquire supplementary information from semantically relevant non-neighboring regions.

[0022] As another technical detail of the present invention, the point suggestion extraction process in the automatic suggestion generation module employs a screening strategy based on non-maximum suppression. (In the probability graph) P In step 1, when selecting the Top-K pixels, for each pixel with the largest response value, the response values ​​of all other pixels within a circular area centered on that pixel and with a radius of R are set to zero, where R is set to 1 / 20 of the image diagonal. This strategy ensures that the generated point cues can uniformly and discretely cover the entire body of the camouflaged target, avoiding excessive clustering of cues in local high-response areas, thereby guiding the mask decoder to form a more complete target perception.

[0023] The scheme described in this invention has been validated on multiple standard camouflage target detection datasets. Experimental data shows that, under zero human prompting, the segmentation mask generated by this invention improves upon existing SAM-based methods in the S-measure structure metric and reduces the mean absolute error (MAE). The self-supervised semantic-guided automatic prompting mechanism and deformable feature fusion strategy described in this invention have significant technical advantages and engineering application value in handling extreme visual camouflage tasks.

[0024] As part of the execution flow of the method described in this invention, the system first receives the image to be detected and scales it to a uniform resolution of 1024×1024. Subsequently, the image is synchronously input to a first feature extractor composed of Transformers and a second feature extractor composed of residual convolutions. The semantic features output by the first feature extractor generate three types of cues—points, boxes, and masks—through an automatic cue generation module, and the cue generation loss is calculated. Simultaneously, the outputs of the first and second feature extractors are aligned and weighted in a deformable spatial channel attention fusion module. The fused feature sequence and the cue output by the cue encoder are embedded in the mask decoder and interact through multiple rounds of cross-attention calculation. Finally, the decoder outputs a pixel-level segmentation probability map, and the final camouflaged target segmentation result is obtained by discrimination using a 0.5 threshold.

[0025] The method described in this invention is not only applicable to camouflaged target detection in static images, but also, due to the good temporal consistency representation capability of its feature extraction branch, can be directly extended to camouflaged target segmentation tasks in video sequences by introducing a temporal dimension memory unit in the mask decoding module. The deformable spatial channel attention fusion module also has a strong compensation effect on boundary blurring caused by motion, further verifying the system integrity and multi-scene adaptability of the technical solution of this invention.

[0026] Finally, the automatic prompt generation module described in this invention can not only generate physical prompts for the current frame segmentation, but its generated intermediate probability map features can also be used as high-dimensional semantic features and injected back into the shallow network of the first feature extractor, forming a recursive feature enhancement path. Under this recursive logic, the model's localization accuracy for camouflaged targets will gradually approach the true value distribution with each iteration. This self-converging technical characteristic is an important feature that distinguishes this invention from existing one-way feedback models. Through the above multi-level and multi-module collaborative operation, this invention constructs a closed-loop, highly automated camouflaged target detection and segmentation system, providing a robust and accurate technical means for target recognition in complex environments. Attached Figure Description

[0027] Figure 1 The diagram shows a comparison of the overall framework of the traditional method and the method of the present invention, where a is a schematic diagram of the traditional process and b is a schematic diagram of the image segmentation method for camouflaged target detection based on DINOv3 and SAM2 of the present invention.

[0028] Figure 2 This is a schematic diagram of the overall structure of the DSP-SAM model (DINOv3 guided prompt generation and SAM2 fusion camouflage target detection network) of the present invention.

[0029] Figure 3 This is a schematic diagram of the DINOv3 automatic prompt generation module of the present invention.

[0030] Figure 4 This is a schematic diagram of the DSAF module of the present invention. Detailed Implementation

[0031] The following is in conjunction with the appendix Figures 1-4 The present invention will be further described in detail below: To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description of the camouflage target detection image segmentation method based on DINOv3 and SAM2, in conjunction with the accompanying drawings and embodiments, is provided. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention.

[0032] In the specific implementation of the camouflage target detection image segmentation method based on DINOv3 and SAM2 described in this invention, see [link to relevant documentation]. Figure 1 and Figure 2 As shown, the entire system architecture aims to guide the segmentation model to accurately lock onto and reconstruct the boundaries of camouflaged targets in extremely low-contrast environments by leveraging the deep semantic priors provided by the self-supervised visual foundation model. In this embodiment, the original image to be processed is first fed into the preprocessing unit, which will process the input image... I The images are uniformly scaled to a resolution of 1024×1024 pixels and then normalized. The processed images are then synchronously input into a parallel dual-branch feature extraction architecture, consisting of a self-supervised visual feature extraction branch and an image structure feature extraction branch.

[0033] The core component of the self-supervised visual feature extraction branch is the first feature extractor. In this embodiment, the extractor adopts the DINOv3 self-supervised pre-trained model based on the Visual Transformer (ViT) architecture. This model learns extremely strong global semantic consistency representations on massive amounts of unlabeled data through a self-distillation mechanism.

[0034] Specifically, the input image is segmented into several 14×14 image patches, which are then linearly projected and fed into a multi-layer self-attention computation unit. To obtain multi-scale semantic features, this embodiment extracts the output from a specific layer of the Transformer backbone network, transposes and resamples it, and generates semantic feature maps with decreasing spatial resolution at four levels. F d1 , F d2 , F d3 , F d4 .

[0035] in, F d1 The dimensions are 256×256, and the number of channels is 256; F d2 The dimensions are 128×128, and the number of channels is 512; F d3 The dimensions are 64×64, and the number of channels is 1024; while the highest layer features F d4 The size is 32×32, and the number of channels is 1024. These feature maps encapsulate the category relevance information of objects against complex backgrounds, enabling the identification of the overall outline of a target through long-range dependencies even when pixel values ​​are extremely close.

[0036] Meanwhile, the second feature extractor in the image structure feature extraction branch employs a hierarchical residual convolutional network or a hierarchical visual encoder. Its primary responsibility is to capture high-frequency structural information of the image, such as edges, textures, and local topological features. The second feature extractor outputs four sets of multi-scale geometric feature maps. F s1 , F s2 , F s3 , F s4 In terms of spatial dimension, it corresponds one-to-one with the output of the first feature extractor 1. Specifically, F s1 Focusing on preserving the fine spatial structure of the original image, this branch can effectively extract the faint physical boundary between the camouflaged target and its adjacent background through 3×3 stride convolution and max pooling operations.

[0037] As a key step in achieving automated segmentation in this invention, the automatic suggestion generation module receives four levels of semantic features from the first feature extractor. This module does not simply stack features, but rather employs a bottom-up semantic enhancement path. First, the highest-level features... F d4 The number of channels was compressed from 1024 to 256 using a 1×1 convolutional layer, and then its spatial resolution was upsampled to 64×64 using bilinear interpolation. F d3 The feature vectors are concatenated along the channel dimension. The concatenated feature vectors are then non-linearly fused using a 3×3 convolutional layer containing residual structures to generate fused features. F m3 The process follows the formula. until the lowest layer of enhanced features is generated. F m1 .

[0038] During the training phase, in order to ensure F mi Including sufficient target localization information, this invention incorporates a lightweight prediction head after each layer of feature fusion. This head consists of 3×3 convolutions, a ReLU activation function, and a 1×1 convolution, used to output a foreground probability map at the corresponding scale. P i These probabilistic graphs, under the supervision of a ground truth mask, learn how to extract the latent distribution of the target from self-supervised semantic features. During the inference phase, the cue extraction logic is automatically executed based on the probabilistic graph P1. First, a fixed threshold is set. τ Binarizing P1 with a value of 0.5 yields a coarse segmentation mask. M prompt If P1 is at a certain coordinate ( x,yIf the response value at a point is greater than 0.5, that point is marked as foreground; otherwise, it is marked as background. Subsequently, a connected component analysis algorithm is used to calculate... M prompt The minimum bounding rectangle of the largest connected region in the middle is used to generate a box hint. B prompt .

[0039] Point hints P points To extract the cues, this invention implements a geometrically constrained non-maximum suppression (NMS) filtering strategy. The system first selects the pixel with the highest response value from P1 as the first seed point. Then, centered on this seed point, within a circular region with a radius R equal to 1 / 20 of the image diagonal, the response values ​​of all pixels are forcibly set to zero. This is to prevent excessive clustering of cue points in highly salient areas of the target (such as the eyes or edges with significant contrast), thereby inducing the generated cue points to uniformly cover the entire camouflaged target. In this embodiment, this process is repeated until K=3 candidate points are extracted, forming a cue point set. This automatically generated cue set enables the SAM2 (SegmentAnythingModel2) architecture to achieve high-quality initial perception without human interaction.

[0040] To achieve deep collaboration across model features, this invention introduces a deformable spatial channel attention fusion module between the two feature extraction branches. For any level of feature fusion, such as the first... i The first layer is the semantic features. F mi Projection onto geometric features F si The same feature space is used. The features are then fed into a deformable spatial attention unit. In this unit, for each spatial location p, the model calculates a two-dimensional offset Δ across N=9 sampling points via a lightweight offset prediction branch. p n Specifically, this invention employs a deformable stripe convolution operator, which decomposes the offset into horizontal and vertical components. This is particularly advantageous for capturing elongated structures or irregular boundaries commonly found in camouflaged targets. Feature values ​​at sampling points are obtained through bilinear interpolation, resulting in the final spatial sampling map. F sample Features and prediction weights of each sampling point m n The weighted sum.

[0041] After obtaining the spatial augmentation features, the module further performs adaptive weighting along the channel dimension. This is achieved through a global average pooling layer. F sampleThe channel attention weights are generated by compressing the channel descriptors and passing them through two fully connected layers. ω Combined with spatial sampling weights, a joint attention map A is generated. The final enhanced features... F fused,i Through the original geometric features F si The result obtained by weighting the residuals is... .

[0042] This design ensures that semantic priors can dynamically guide the model to find geometric edges that conform to semantic logic in complex textured backgrounds, fundamentally solving the problem of uncertain pixel classification of camouflaged target edges.

[0043] The mask decoding module is responsible for fusing the multi-scale features. F fused,i The final fusion is performed with the automatically generated cue embeddings. The cue encoder first transforms point cue, bounding box cue, and mask cue into cue embeddings of a unified dimension. Point and bounding box coordinates are encoded using positional encoding based on sine and cosine functions, mapping them to a 256-dimensional vector space. The mask cue is downsampled using four 3×3 convolutional layers to ensure its scale matches the fused features. The mask decoder integrates multiple bidirectional cross-attention layers, allowing the cue embeddings to search for target-related activation values ​​within the feature embeddings, and also allowing the feature embeddings to adjust their response distribution based on the cue information. After two rounds of interaction, the decoder uses transposed convolutional layers to gradually restore the feature map resolution from 256×256 to 1024×1024, finally outputting a pixel-level segmentation probability map through a sigmoid function. S p .

[0044] In the system optimization process, the multi-task joint loss function adopted in this invention L total It played a crucial role. To address the challenge of defining the difficult-to-define edges of camouflaged targets, the weighted binary cross-entropy loss... L wbce Spatial weighting factor was introduced. w ij This factor calculates the difference between each pixel and its neighborhood mean by performing a 31×31 average pooling on the ground truth mask. For pixels in the boundary region, w ij The value of this will increase significantly, thus forcing the model to pay more attention to these error-prone areas during training. Similarly, the weighted intersection-union loss... L wiou Spatial weights were also considered, and the recall rate of the model for small-scale camouflaged targets was improved by weighting the intersection and union.

[0045] This invention improves the S-measure metric compared to the basic SAM model. The self-supervised semantic features guided by DINOv3 provide the model with a more robust target localization capability than simple geometric features. Regarding the MAE metric, this invention significantly reduces the mean absolute error. The deformable spatial channel attention fusion module designed in this invention can greatly reduce the misclassification rate of camouflaged edges.

[0046] In specific implementation scenarios, this invention not only demonstrates extremely high static segmentation accuracy but also exhibits strong generalization robustness. As a specific embodiment, when the system processes an image of a gecko hidden in a pile of dead leaves, the first feature extractor, through self-supervised learning of object integrity knowledge, in... F d4 The basic structure of the gecko's torso has been outlined, even though its skin color is completely indistinguishable from the dead leaves. Based on this semantic information, the automatic prompting module generates three prompts, accurately distributed around the gecko's head, tail, and the ends of its limbs, avoiding the surrounding dead leaves with similar textures. Subsequently, the deformable spatial channel attention fusion module adaptively adjusts the sampling range through striped convolution, precisely capturing the tiny gaps between the gecko's toes—something difficult to achieve with traditional fixed convolution kernels.

[0047] Furthermore, as an extended embodiment of the present invention, when processing camouflaged targets in a video sequence, the mask decoding module is configured to include a Convolutional Long Short-Term Memory (ConvLSTM) unit. This unit stores the cue embedding vector and feature distribution generated in the previous frame. In the processing of the current frame, the system downsamples the predicted mask from the previous frame and uses it as an additional "memory cue" input. This allows the model to maintain segmentation stability by utilizing temporal continuity even when the target shifts, causing instantaneous boundaries to become more blurred. This spatiotemporal collaborative mechanism enables the present invention to maintain a high S-measure metric when processing dynamic camouflage segmentation tasks.

[0048] In this invention, the intermediate probability map features generated by the automatic suggestion generation module also play a recursive enhancement role in the system's deep loop. Specifically, the highest resolution foreground probability map generated in the first round of inference is used as the basis for further analysis. P 1, as an additional feature channel, is concatenated back into the shallow features of the first feature extractor and used for a second forward computation. Through recursive feature refinement, the model's extraction accuracy for slender appendages (such as insect antennae or animal hair) can be further improved. This self-converging architecture ensures that the system maintains a very high segmentation success rate even in highly challenging camouflage scenarios, such as when the target-background contrast is less than 5%.

[0049] In another embodiment, see Figure 2 This invention utilizes the overall structure of the DSP-SAM model, comprising four main components: a DINOv3 backbone and an automatic prompting generation module (PromptGenerator), a SAM2 image encoder, a deformable spatial channel attention fusion module (DSA), and a SAM2 prompt encoder and mask decoder. Specifically, the input image F = { f 1, f 2, f 3, f 4} First, the image encoders DINOv3 and SAM2 are fed in parallel. DINOv3 extracts semantic features at four scales, and the automatic prompt generation module obtains fused features at three scales through layer-by-layer upsampling, concatenation, and convolution. During the training phase, ground truth masks are used for supervision to enhance the response to camouflaged regions. Then, the generator outputs three types of cues for SAM2 to use based on the fused features: point cues, bounding box cues, and mask cues. At the same time, the SAM2 image encoder extracts four-level features to preserve local structural information. Subsequently, at the feature level, the corresponding scale features of DINOv3 and SAM2 are fused through a deformable spatial channel attention fusion module. First, the DINO features are projected onto a channel space aligned with SAM2, and deformable convolution is used to capture cross-positional responses and adaptively weight them in the channel dimension to achieve high-quality semantic injection into geometry. Finally, the prompt generated by the automatic prompt generation module is fed into the prompt encoder of SAM2 to obtain the prompt embedding. The mask_decoder generates the final mask output under the condition of receiving the fused multi-scale features and prompt embedding. This design enables DINOv3's high-level semantic capabilities to both directly guide SAM2 in the form of cues and effectively supplement the structural details of SAM2 at the feature level through DSA, thereby achieving complementarity and synergy between semantic localization and boundary recovery of camouflaged targets.

[0050] In camouflaged target detection, SAM2 relies on manually provided point, bounding box, or coarse mask cues to achieve reliable performance, which has significant limitations in complex scenes where humans struggle to distinguish camouflaged areas. For this purpose, see [link to relevant documentation]. Figure 3 As shown, this embodiment designs an automatic prompt generation module based on DINOv3, which replaces manual interaction with a semantic-driven approach, enabling SAM2 to maintain stable segmentation capabilities without manual prompts. Specifically, firstly, the multi-scale features of DINOv3 are... Upsampling is performed level by level and aligned with shallow features to construct a semantically consistent fusion representation, in the form of: , And obtain the enhanced features through convolution. .

[0051] During the prompt generation phase, for the enhanced DINO characteristics First, a lightweight FCN is used to predict pixel-level foreground probability maps: ; Where S(·) represents the Sigmoid activation function. This prediction result characterizes the probability distribution of each pixel belonging to the camouflaged target region. Subsequently, the predicted mask is binarized using a fixed threshold to obtain a coarse segmentation mask: ; This mask map serves both as a mask hint for SAM2 and as a spatial prior for the subsequent generation of point and box hints.

[0052] After obtaining the binary mask, a spatial scan is performed on the foreground region of each image to extract the set of coordinates of all foreground pixels: .

[0053] like If not empty, the target bounding box is generated by calculating its minimum bounding rectangle. (Hint:) , in ; When the predicted mask is empty, the box cue is set to a zero vector to ensure computational stability. This method allows the generated box cue to closely fit the spatial extent of the camouflaged target, providing greater adaptability compared to fixed-scale or heuristic boxes.

[0054] To further provide high-confidence local guidance information, the K pixels with the highest response values ​​are selected as point cues from the predicted probability mask. Specifically, after flattening the probability mask, the Top-K positions with the highest responses are selected: ; These points correspond to the regions where the model is most confident they belong to the foreground, effectively guiding the SAM2 cue encoder to focus on the core regions of the camouflaged target. Compared to random or rule-based sampling, this strategy exhibits stronger semantic consistency and robustness.

[0055] Ultimately, dot hints, box hints, and mask hints from different scales were integrated into a unified hint set. The data is then fed into the SAM2 prompt encoder to generate prompt embeddings. Leveraging the semantic advantages of DINOv3, the automatic prompt generation module can automatically discover potential semantic anomaly patterns in camouflaged regions, fundamentally reducing SAM2's reliance on manual prompts. This gives SAM2 greater autonomy and generalization capabilities in camouflaged scenarios, achieving a key paradigm shift from "manual prompt-driven segmentation" to "semantic automatic prompt-driven segmentation."

[0056] In the camouflage target detection task, the features extracted by DINOv3 and SAM2 differ significantly in their representational emphasis: DINOv3's self-supervised Transformer features exhibit strong global semantic consistency, effectively highlighting the semantic response of potentially camouflaged regions; while the SAM2 image encoder excels at modeling local geometric structures and contour information, but is susceptible to semantic confusion in weak-contrast and weak-texture regions. To fully leverage their complementary advantages and achieve effective cross-model feature alignment, see [reference needed]. Figure 4 As shown, a deformable spatial attention fusion module (DSAF) was designed to finely modulate SAM features in a semantically driven manner. Specifically, let the semantic features extracted by DINOv3 be: The geometric features output by the SAM2 image encoder are: First, the DINO features are projected to the same channel dimension as the SAM features using an 11×11 convolution to achieve channel alignment: ; Then, the aligned DINO features The deformable spatial-channel attention fusion module (DSA) is used to generate a semantically guided spatial-channel joint attention graph. ; The DSA module is a lightweight deformable feature modeling structure inspired by the intrinsic connection between DCNv3 and spatial attention mechanisms. Unlike traditional deformable convolutions, DSA employs Deformable Separable Convolution Network (DSCN) as its core operator, decoupling two-dimensional deformable sampling into two one-dimensional irregular samplings along the x and y axes. It also uses linear interpolation instead of bilinear interpolation, significantly reducing computational complexity and the number of parameters. In its implementation, DSA consists of a pair of DSCN operations along the x and y directions, a 5×5 depthwise separable convolution, two 1×1 channel projections, and a GELU activation function. Through residual connections and spatial gating mechanisms, it achieves joint modeling of responses across space and channels. This structure inherits the global irregular sampling capabilities of DCNv3 while maintaining excellent lightweight characteristics, enabling it to adaptively focus on key regions with irregular shapes and blurred boundaries in camouflaged targets. The specific process of DSA is as follows: ; The resulting attention map A is used as semantic modulation weights to perform element-wise weighting of the SAM features. Finally, a 3×3 convolution is used to further integrate and smooth the fused features to enhance local consistency and improve feature expressiveness. ; Through the above design, the DSAF module implements a deformable spatial attention modulation mechanism driven by DINOv3 semantic prior, enabling the geometric features of SAM2 to adaptively focus on potential camouflage regions under semantic guidance. This effectively alleviates the segmentation difficulties caused by weak boundaries and high semantic confusion, thereby providing a more discriminative feature representation for the subsequent decoder to generate high-quality camouflage target masks.

[0057] By using a combination of weighted binary cross-entropy (BCE) loss and weighted intersection-over-union (IoU) loss as the basic training objective, this scheme can effectively balance pixel-level accuracy and structural consistency in camouflaged object detection.

[0058] The weighted BCE and IoU losses are expressed as follows: ; Where P represents the ground truth mask, S is the prediction map, P=σ(S), and w(x,y) is the spatial weight of the region emphasizing information.

[0059] For each segmentation prediction S, the structure-aware loss is defined as: ; In DSP-SAM, the loss function applies to both the decoder output {Sp} and the DINOv3-guided auto-hint generation module output {Si}. The overall training objective can be expressed as: .

[0060] In summary, the camouflaged target detection image segmentation method based on DINOv3 and SAM2 described in this invention successfully achieves autonomous identification and accurate segmentation of camouflaged targets in complex environments by constructing a sophisticated dual-branch feature extraction system, utilizing the semantic generalization capability of a self-supervised visual model, and combining it with automated prompt generation logic and deformable feature fusion strategies. In engineering practice, it not only eliminates reliance on manual intervention and significantly reduces application costs, but also sets new industry standards in key metrics such as segmentation accuracy and boundary reconstruction. The various modules proposed in this invention, including the application of deformable striped convolution, point prompt extraction logic with NMS constraints, and the combined optimization of weighted loss functions, together constitute a closed-loop and efficient engineering solution with high promotional value and practical application significance.

[0061] For those skilled in the art, any equivalent substitutions made to the parameter configurations, convolutional layer depths, or specific backbone networks of the feature extraction branches of the aforementioned modules without departing from the core technical principles of this invention should be included within the scope of protection of this invention. The systematic nature, completeness, and superiority of the solution described in this invention, supported by massive amounts of experimental data, provide solid technical support for its quantitative implementation in fields such as military reconnaissance, wildlife protection, and disaster search and rescue.

Claims

1. A method for image segmentation of camouflaged target detection based on DINOv3 and SAM2, characterized in that, The method is implemented through an architecture of automatic prompt generation and multi-scale feature fusion guided by a self-supervised visual base model. The architecture consists of a self-supervised visual feature extraction branch, an image structure feature extraction branch, an automatic prompt generation module, a deformable spatial channel attention fusion module, and a mask decoding module working collaboratively. The method includes the following steps: S1: The input image to be processed is synchronously input into the self-supervised visual feature extraction branch and the image structure feature extraction branch; the first feature extractor in the self-supervised visual feature extraction branch extracts a four-level multi-scale semantic feature map with global semantic consistency, denoted as follows: F d1 , F d2 , F d3 , F d4 Four sets of multi-scale geometric feature maps are obtained in parallel through the second feature extractor in the image structure feature extraction branch, denoted as follows: F s1 , F s2 , F s3 , F s4 ; S2: Input the multi-scale semantic feature map into the automatic suggestion generation module, and generate multi-scale fused features through a bottom-up semantic enhancement path. F mi , i =1,2,3, and a pixel-level foreground probability map generated by the automatic prompt generation module based on the lowest-level enhanced features, automatically executes the prompt extraction logic, and outputs a masked prompt. M prompt Box prompt B prompt And tips P points An automatically generated set of prompts; S3: A deformable spatial channel attention fusion module is used to perform fine-grained cross-model feature alignment of multi-scale semantic feature maps and multi-scale geometric feature maps. Spatial sampling adaptive offset is performed through deformable stripe convolution operator, and enhanced features are generated by combining channel adaptive weighting mechanism. F fused,i ; S4: Input the automatically generated cue set into the cue encoder in the mask decoding module to convert it into a cue embedding vector, and then combine the cue embedding vector with the enhanced features. F fused,i The data are fed into the mask decoder together; the mask decoder uses a multi-layer bidirectional cross-attention mechanism to dynamically exchange information, and finally restores the resolution and outputs a pixel-level segmentation mask through transposed convolution; The specific logic of the auto-completion generation module executing the bottom-up semantic enhancement path is as follows: First, the highest-level features F d4 After 1×1 convolution compression, the signal is upsampled using bilinear interpolation. F d3 Spatial scale, and with F d3 The concatenation is performed along the channel dimension, and then fused using a 3×3 residual convolutional layer to obtain the fused features. F m3 ; Through iterative formula Generate sequentially F m2 and F m1 ; During the training phase, each layer fuses features. F mi Each layer is connected to a lightweight fully convolutional layer to predict pixel-level foreground probability maps at the corresponding scale. Furthermore, the truth mask is constrained by a deep supervised loss function to ensure that the automatic prompt generation module can accurately locate the potential salient area where the disguised target is located.

2. The image segmentation method for camouflaged target detection based on DINOv3 and SAM2 according to claim 1, characterized in that, The first feature extractor uses a self-supervised pre-trained model based on the visual Transformer architecture, and its feature extraction process includes: The input image with a resolution of 1024×1024 is divided into several fixed-size image blocks, which are then mapped to a high-dimensional embedding space through a linear projection layer and positional encoding is added. Long-range dependencies of images are extracted using multi-layer self-attention computing units, and feature sampling and resampling are performed at different depth levels to output four-level semantic feature maps with downsampling ratios of 4, 8, 16, and 32. Among them, the first-level semantic feature map F d1 The size is 256×256, fourth-level semantic feature map F d4 The dimensions are 32×32; The first feature extractor, through discriminative representations obtained by pre-training on large-scale unsupervised datasets, maintains a stable response to the semantics of the target subject in scenarios where the contrast between the camouflaged region and the background is below a preset threshold.

3. The image segmentation method for camouflaged target detection based on DINOv3 and SAM2 according to claim 1, characterized in that, The second feature extractor employs a hierarchical visual encoder, which captures high-frequency components of the input image through layer-by-layer convolution operations and spatial downsampling operations, thereby extracting multi-scale geometric feature maps. F si ; Multiscale geometric feature map F si Spatial resolution and multi-scale semantic feature maps F di Layer-by-layer correspondence is used to characterize the local geometric structure, texture information, and fine-grained edge features of the camouflaged target, in order to make up for the lack of local topological structure characterization by the first feature extractor.

4. The camouflaged target detection image segmentation method based on DINOv3 and SAM2 according to claim 1, characterized in that, The hint extraction logic includes the following steps: Mask hint generation: Set a fixed threshold τ =0.5 Binarize the foreground probability map P1 to generate a coarse segmentation mask as a mask cue. M prompt ; Box tooltip generation: for masked tooltips M prompt Perform a spatial scan of the connected regions in the code, calculate the minimum bounding rectangle of the connected regions, and generate bounding box hints. B prompt ; like M prompt If it is an empty set, then... B prompt Set it to a vector of all zeros; Point hint generation: A nonmaximum suppression screening strategy based on geometric constraints is employed, based on the foreground probability map. Filter by the highest response value The pixel coordinates are used as a point hint. P points During the iterative selection process, for each pixel with the largest current response value selected as a seed point, the response values ​​of all other pixels within a circular region centered on that seed point and with radius R are forcibly set to zero, until K points are selected; where the radius... The input image is set to 1 / 20 of its diagonal length. The distribution range indicated by the scattered points guides the mask decoding module to perceive the entire disguised target.

5. The camouflaged target detection image segmentation method based on DINOv3 and SAM2 according to claim 1, characterized in that, Deformable Space Channel Attention Fusion Module for the first The calculation process for fusing layer features is as follows: semantic features F mi Projecting the geometric features through a 1×1 convolution F si With the same channel dimension, projection features are obtained. F mi ' Projection features F mi ' The data is fed into a deformable spatial attention unit, which uses a predictive branch to calculate the offset Δ of N sampling points. p n and the corresponding sampling weights m n ; The deformable spatial attention unit employs a deformable fringe convolution operator to decompose the two-dimensional offset into one-dimensional irregular samples in the horizontal and vertical directions. It then uses linear interpolation to obtain feature values ​​at non-integer coordinates, generating a spatial sampling map. F sample The calculation formula is: ； in W n This is a learnable weight matrix.

6. The image segmentation method for camouflaged target detection based on DINOv3 and SAM2 according to claim 5, characterized in that, The deformable spatial channel attention fusion module also includes a channel adaptive weighting mechanism: Spatial sampling map is extracted using a global average pooling layer. F sample The channel statistics are used to calculate the channel attention weights through two fully connected layers and a sigmoid activation function. Spatial sampling weights and channel attention weights ω Perform element-wise multiplication to construct a joint attention graph A; Through formula Calculate the enhanced features F fused,i , where ⊙ represents element-wise weighting; The deformable spatial channel attention fusion module guides the redistribution of geometric features through semantic priors, enabling the model to spontaneously focus on the weak boundary regions of camouflaged targets.

7. The camouflaged target detection image segmentation method based on DINOv3 and SAM2 according to claim 1, characterized in that, The interaction process of the mask decoding module is as follows: Encoder point prompt P points The coordinates are encoded using sine and cosine positions, as indicated by the box prompt. B prompt The coordinate information is encoded using linear projection, and the mask is used for hints. M prompt Multi-layer downsampling convolutional coding is used to uniformly transform the cue embedding vectors into a high-dimensional space; The bidirectional cross-attention mechanism in the mask decoder performs multiple rounds of iterative information interaction between the feature embedding and the cue embedding vector. It corrects the response distribution in the feature map through cue information, and restores the feature map to the original image resolution through a transposed convolutional layer in the last layer of the decoding process. Finally, the final segmentation mask is output through the Sigmoid function mapping.

8. The image segmentation method for camouflaged target detection based on DINOv3 and SAM2 according to claim 1, characterized in that, The method employs a multi-task joint loss function. L total Training, joint loss function L total It is a weighted sum of the decoder output loss and the auto-completion generation module output loss, and its expression is: ； Where G is the truth mask. S p This is the final output segmentation probability map. S i λ is the intermediate probability plot generated by the auto-completion module. i The hyperparameters are preset; weighted binary cross-entropy loss. L wbce Weighted intersection and comparison loss L wiou Spatial weighting factors were introduced in all of them. w ij， Spatial weighting factor w ij This is achieved by performing average pooling on the ground truth mask and calculating the difference between the central and edge regions. The aim is to increase the loss weight of pixels in the boundary region of the camouflaged target, thereby enhancing the model's ability to capture weak boundaries.

9. The image segmentation method for camouflaged target detection based on DINOv3 and SAM2 according to claim 1, characterized in that, The method also includes a recursive feature enhancement path: The highest resolution foreground probability map generated by the first round of forward inference is used as a high-dimensional semantic feature, which is then injected backward and concatenated into the shallow network of the first feature extractor. Through iterative calculation, the model's localization accuracy for camouflaged targets gradually converges. Meanwhile, for the task of detecting camouflaged targets in video sequences, a temporal memory unit is introduced into the mask decoding module. The cue embedding and feature distribution of the preceding frame are stored through a convolutional long short-term memory network. The temporal continuity is used to compensate for the boundary blurring caused by motion, so as to achieve spatiotemporal collaborative camouflaged target segmentation.

Images