A dynamic multi-scale salient object detection method based on a SAM large model

Abstract

The application provides a dynamic multi-scale salient object detection method based on a SAM large model, and relates to the technical field of computer vision and image processing. First, a network model is constructed based on a SAM basic framework, specifically including an image encoder, a mask decoder, a multi-layer feature cross fusion module MFCM and a detail enhancement module DEM; then, a to-be-detected image is input into the encoder, dynamic multi-scale features are adaptively extracted by using a DMSA, high-level semantics and low-level details are deeply interactively fused based on a cascaded cross attention mechanism of the MFCM; finally, in the decoding stage, the fused features are input into the DEM, a high-frequency subband in a frequency domain is extracted by using a discrete wavelet transform to restore edge structure details, and a prediction mask of a salient object is generated; and the network model is trained according to a loss function and is used for salient object detection.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, and in particular to a dynamic multi-scale salient target detection method based on the SAM large model. Background Technology

[0002] Salient Object Detection (SOD) aims to identify and segment the most prominent objects in an image. As a fundamental vision task, it plays a crucial role in many downstream applications such as object tracking, scene segmentation, and person re-identification. In recent years, methods based on convolutional neural networks and Transformers have made significant progress in salient object detection, but they still face the problem of poor generalization ability in complex scenes.

[0003] SegmentAnythingModel (SAM) is a powerful and general-purpose visual model for image segmentation with excellent generalization capabilities. However, SAM has the following limitations in salient object detection tasks: First, SAM heavily relies on manual cues (such as points and boxes) and cannot be directly used for fully automated salient object detection; second, fine-tuning all parameters of SAM is computationally expensive and prone to catastrophic forgetting.

[0004] To address these issues, existing research typically employs parameter fine-tuning strategies, transferring the SAM to the SOD task through adapter insertion. However, existing adaptation strategies still suffer from three inherent drawbacks: (1) static adapters lack dynamic adaptability and struggle to handle targets with vastly different sizes; (2) simple feature concatenation strategies ignore cross-level semantic relevance; and (3) the decoding process is limited to spatial domain operations, inevitably discarding high-frequency structural details. These limitations result in existing methods often outputting incomplete masks with blurred boundaries when processing multi-target or complex textured scenes. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a dynamic multi-scale salient target detection method based on a large SAM model. This method, by introducing dynamic multi-scale perception and frequency domain detail enhancement mechanisms, effectively solves the problems of incomplete salient target masks and blurred boundaries in complex scenes while maintaining the generalization ability of the base model.

[0006] The technical solution adopted in this invention is as follows: A dynamic multi-scale salient object detection method based on the SAM large model includes the following steps: Step 1: Construct the network model; the network model is based on the SAM architecture and specifically includes an image encoder, a mask decoder, a multi-layer feature cross-fusion module (MFCM), and a detail enhancement module (DEM); The image encoder receives the image to be detected and extracts multi-level image features; the multi-level feature cross-fusion module (MFCM) is connected to the image encoder, receives and fuses the multi-level image features, and outputs fused features; the mask decoder receives the fused features and outputs high-level semantic features and intermediate coarse prediction masks; the detail enhancement module (DEM) is connected to both the image encoder and the mask decoder, and combines high-level semantic features, deep encoder features, and local features of the input image to generate a prediction mask for the final salient target.

[0007] The image encoder consists of several Transformer blocks, each containing a multi-head self-attention (MHSA), a multilayer perceptron (MLP), and a layer normalization (LN). The image encoder embeds a dynamic multi-scale adapter (DMSA). Specifically, the dynamic multi-scale adapter (DMSA) is embedded before the first layer normalization operation of each Transformer block of the image encoder. The Multi-Level Feature Cross-Fusion Module (MFCM) receives multi-level image features output by the image encoder and achieves the fusion of high-level semantics and low-level details through a cascaded cross-attention mechanism, outputting fused features. The detail enhancement module (DEM) includes a main decoding branch and an auxiliary detail branch. The main decoding branch receives high-level semantic features output by the mask decoder and deep encoder features output by the image encoder, concatenates them, and upsamples them progressively to output a coarse backbone feature. The auxiliary detail branch receives local features of the input image, extracts and enhances high-frequency subbands through discrete wavelet transform, and outputs frequency-domain enhanced detail features. The coarse backbone feature and the frequency-domain enhanced detail feature are finally concatenated and combined to generate a prediction mask.

[0008] Step 2: Input the image to be detected into the image encoder, and use the Dynamic Multi-Scale Adapter (DMSA) to perform dynamic multi-scale feature extraction to obtain multi-level image features; Specifically, the process involves: inputting the image to be detected into an image encoder, converting it into initial image features through block embedding; during the image encoder's processing, inputting the current image features into a Dynamic Multi-Scale Adapter (DMSA), first calibrating and reducing the dimensionality of the current image features using learnable scale and displacement parameters; then, using multiple parallel deep convolutional branches with different receptive field sizes within the DMSA, extracting multi-scale spatial features from the dimensionality-reduced image features; then, dynamically weighting and fusing the different scale features output by the multiple parallel deep convolutional branches using learnable weights to obtain multi-scale fused features; subsequently, optimizing the multi-scale fused features through residual structures and activation functions, and restoring the number of channels of the optimized multi-scale fused features to their original state before dimensionality reduction using a linear upprojection layer. Dimensions; Finally, the features after restoring the original dimensions are added to the image features before inputting DMSA through residual connection, and the result is output as the processing result of the current Transformer block; The initial image features are processed sequentially by multiple Transformer blocks with DMSA in the image encoder, and multiple features of different depths are output. These features of different depths together constitute the multi-level image features; Specifically, the layer number of the Transformer block in the image encoder is used as an objective metric, and the layer number comparison rule is set: Among the multiple features of different layers extracted, the features with smaller layer numbers, that is, those ranked before the set relative layer number threshold, are defined as low-level features containing low-level detailed information; the features with larger layer numbers, that is, those ranked after the set relative layer number threshold, are defined as high-level features containing high-level semantic information. Given input features , No. i The operation formulas for a Transformer block are as follows: (1); (2); in, MHSA , MLP , LN These represent multi-head self-attention, multilayer perceptron, and layer normalization, respectively. This represents the intermediate transition features obtained after multi-head self-attention and residual connections. Indicates the first i The final output features of each Transformer block; The calculation process of the Dynamic Multiscale Adapter (DMSA) is as follows: First, the input features are processed... The distribution is calibrated by projecting it onto a low-dimensional subspace through a learnable linear layer: (3); in, α and β These are the learnable scaling and offset parameters, respectively. D i Indicates the first i Linear projection layers used for dimensionality reduction in each adapter; Subsequently, three parallel depthwise convolutional branches with kernel sizes of 3×3, 5×5, and 7×7 were used to extract multi-scale features, and learnable weights were applied. w k Dynamic fusion is performed to obtain multi-scale fusion features F. ms : (4); in, DWConv k Indicates the kernel size as k × k deep convolutional layers, It is a minimal constant used for numerical stability.

[0009] Next, lightweight residual blocks are introduced to optimize the fused features: (5); Among them, F ref This represents the optimized multi-scale fusion feature. Represents the GELU activation function. Conv 1×1 This represents a convolutional layer with a kernel size of 1×1.

[0010] Subsequently, 1×1 convolutions are used to compress and integrate the optimized multi-scale fusion features, and the GELU activation function is used again for non-linear activation. The output is then projected back to the original dimension through another linear layer. Finally, the output of the dynamic multi-scale adapter DMSA is obtained through residual connections. (6); in, This represents the final output feature. U i This indicates a linear projection layer.

[0011] Step 3: Input the multi-level image features into the Multi-Level Feature Cross-Fusion Module (MFCM). Through a cascaded top-down cross-attention mechanism, the high-level semantics and low-level details are fused to obtain the fused features. Specifically, the following steps are taken: the current low-level features are recalibrated using channel attention; cross-attention is calculated using the calibrated current low-level features as the query and the high-level features as the key and value; and the multi-level image features are fused, updated, and stitched together in a top-down cascade manner.

[0012] Top-down cascade fusion is based on multi-layer cross-attention MCA.

[0013] First, channel attention is applied to the input low-level features for recalibration: (7); in, This indicates the current low-level features after channel recalibration. δ Indicates linear projection. GAP This is global average pooling.

[0014] Next, the calibrated low-level features are used as the query, and the higher-level features are used as the key and value for cross-attention calculation: (8); (9); (10); in, Q , K , V These represent the query vector, key vector, and value vector, respectively. Attention ( ) indicates cross-attention calculation. Softmax ( ) represents the Softmax normalized activation function, with the superscript of the parameter. T This indicates that the matrix is ​​transposed. BN Indicates batch normalization. This represents the current feature updated after incorporating high-level semantics.

[0015] The entire calculation process of cross-attention, combining equations (7) to (10), is collectively referred to as multi-layer cross-attention calculation, denoted as the MCA() function. For the deepest feature... Since there is no higher-level semantic reference, only channel attention optimization is performed: (11); The remaining levels are cascaded in a top-down order: (12); Finally, all the refined features are concatenated and processed by convolution to generate fused features. For use by the decoder: (13); in, Concat This indicates that feature splicing is performed along the channel dimension.

[0016] Step 4: Input the fused features into the detail enhancement module DEM, use Discrete Wavelet Transform (DWT) to extract and enhance high-frequency structural details, and generate a prediction mask for salient targets; Specifically, the fused features are input into the mask decoder, and after processing, the mask decoder outputs high-level semantic features. Simultaneously, the mask decoder uses these fused features to directly output an intermediate coarse prediction mask S. m Next, the deepest features from the last layer output of the image encoder are extracted and upsampled to serve as deep encoder features. In the main decoding branch, high-level semantic features are concatenated with deep encoder features and progressively upsampled to obtain the backbone coarse features. In the auxiliary detail branch, wavelet attention (WA) is introduced, and the local features of the input image are separated into low-frequency sub-bands and multiple high-frequency sub-bands through discrete wavelet transform (DWT). Then, a learnable network parameter threshold is applied to the high-frequency sub-bands, and the high-frequency sub-bands after applying the network parameter threshold are combined with the low-frequency sub-bands. Edge enhancement features are reconstructed through inverse discrete wavelet transform (IDWT) to generate frequency domain enhanced detail features. Finally, the backbone coarse features and frequency domain enhanced detail features are concatenated and combined to generate the final prediction mask.

[0017] In the main decoding branch, the high-level semantic features of the mask decoder are... Features of deep encoders After splicing and compressing the channels, multiple bilinear interpolation upsampling operations are performed to restore spatial resolution. (14); (15); in, This represents the channel compression features obtained by 1×1 convolution dimensionality reduction after concatenation. Indicates the general characteristics of the main trunk. US ×2 This indicates a 2x bilinear interpolation upsampling. Conv 3×3 This indicates a convolutional layer with a kernel size of 3×3.

[0018] In the auxiliary detail branch, wavelet attention (WA) is used to extract frequency domain details; firstly, local features F of image I are extracted. local : (16); Next, the local features F are transformed using Discrete Wavelet Transform (DWT). local It is decomposed into one low-frequency sub-band L and three high-frequency sub-bands, specifically including vertical... ,level ,diagonal : (17); A learnable threshold is applied to the high-frequency subbands and they are spliced ​​and fused. Then, the low-frequency subbands are combined, and edge enhancement features are reconstructed through inverse discrete wavelet transform (IDWT). (18); in, IDWT ( ) denotes the inverse discrete wavelet transform operation. This represents the edge enhancement features obtained after reconstruction via inverse wavelet transform.

[0019] The wavelet attention weights are generated and multiplied by the original input to obtain the frequency domain enhanced detail features. : (19); in, Flatten ( ) indicates the feature flattening operation; Finally, the main trunk coarse features F up With frequency domain enhancement detail features F wa Concatenate the data and output the final prediction mask S. f : (20); Step 5: Train the network model based on the loss function, and use the trained network model to detect salient targets in the image to be detected, and obtain the final prediction mask.

[0020] Step 5 is specifically divided into a model training phase and a model testing phase: During the model training phase: Obtain training image samples with ground truth mask labels, input them into the network model, and process them sequentially through steps 2 to 4; obtain the final output predicted mask S. f and the intermediate coarse prediction mask S output in step 4 m The loss function specifically includes the BCE loss. IoU loss and L1 loss ; respectively in the final refined output prediction mask S f and the intermediate coarse prediction mask S m Calculate the loss above, total loss function for: (twenty one); (twenty two); Where G represents the true mask label of the salient target, and S represents the predicted mask. This represents the loss calculation function between the predicted mask and the true label. This indicates that the final prediction mask Sf Calculated loss, This indicates that the intermediate coarse prediction mask S m Calculated loss.

[0021] During the model detection phase: The actual image to be detected is input into the trained network model, and the processing flow of steps 2 to 4 above is executed in sequence. The prediction mask finally output by the DEM is the detection result of the salient target in the image to be detected.

[0022] The beneficial effects of adopting the above technical solution are as follows: This invention provides a dynamic multi-scale salient target detection method based on the SAM large model, which has the following beneficial effects: (1) A dynamic multi-scale adapter (DMSA) is introduced: by using learnable dynamic weights across multiple receptive fields, multi-scale features are adaptively calibrated, which solves the problem of lack of dynamic adaptability of existing static adapters, improves the detection accuracy of targets of various sizes, and only adds a very small number of parameters.

[0023] (2) A multi-layer feature cross-fusion module (MFCM) was designed: a cascaded top-down cross-attention mechanism was adopted to replace the simple splicing fusion, which promoted deep interaction between high-level semantics and low-level details, effectively filtering out background noise and highlighting salient objects.

[0024] (3) A detail enhancement module (DEM) was developed: Discrete wavelet transform (DWT) is used to explicitly isolate and enhance high-frequency subbands in the frequency domain. This mechanism successfully recovers sharp boundaries and complex textures that are often lost in spatial domain operations, greatly improving the model's segmentation accuracy for complex edge details. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall architecture of the dynamic multi-scale salient target detection method based on the SAM large model provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the structure of the Dynamic Multi-Scale Adapter (DMSA) in an embodiment of the present invention; Figure 3 This is a schematic diagram of the calculation process of multi-layer cross attention (MCA) in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the detail enhancement module (DEM) in an embodiment of the present invention. Detailed Implementation

[0026] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0027] Example 1: A dynamic multi-scale salient object detection method based on the SAM large model, such as Figure 1 As shown, it includes the following steps: Step 1: Construct the network model; the network model is based on the SAM architecture and specifically includes an image encoder, a mask decoder, a multi-layer feature cross-fusion module (MFCM), and a detail enhancement module (DEM); The image encoder receives the image to be detected and extracts multi-level image features; the multi-level feature cross-fusion module (MFCM) is connected to the image encoder, receives and fuses the multi-level image features, and outputs fused features; the mask decoder receives the fused features and outputs high-level semantic features and intermediate coarse prediction masks; the detail enhancement module (DEM) is connected to both the image encoder and the mask decoder, and combines high-level semantic features, deep encoder features, and local features of the input image to generate a prediction mask for the final salient target.

[0028] The image encoder consists of several Transformer blocks, each containing a multi-head self-attention (MHSA), a multilayer perceptron (MLP), and a layer normalization (LN). The image encoder embeds a dynamic multi-scale adapter (DMSA). Specifically, the dynamic multi-scale adapter (DMSA) is embedded before the first layer normalization operation of each Transformer block of the image encoder. The Multi-Level Feature Cross-Fusion Module (MFCM) receives multi-level image features output by the image encoder and achieves the fusion of high-level semantics and low-level details through a cascaded cross-attention mechanism, outputting fused features. The detail enhancement module (DEM) includes a main decoding branch and an auxiliary detail branch. The main decoding branch receives high-level semantic features output by the mask decoder and deep encoder features output by the image encoder, concatenates them, and upsamples them progressively to output a coarse backbone feature. The auxiliary detail branch receives local features of the input image, extracts and enhances high-frequency subbands through discrete wavelet transform, and outputs frequency-domain enhanced detail features. The coarse backbone feature and the frequency-domain enhanced detail feature are finally concatenated and combined to generate a prediction mask.

[0029] Step 2: Input the image to be detected into the image encoder, and use the Dynamic Multi-Scale Adapter (DMSA) to perform dynamic multi-scale feature extraction to obtain multi-level image features; Specifically, the process involves: inputting the image to be detected into an image encoder, converting it into initial image features through block embedding; during the image encoder's processing, inputting the current image features into a Dynamic Multi-Scale Adapter (DMSA), first calibrating and reducing the dimensionality of the current image features using learnable scale and displacement parameters; then, using multiple parallel deep convolutional branches with different receptive field sizes within the DMSA, extracting multi-scale spatial features from the dimensionality-reduced image features; then, dynamically weighting and fusing the different scale features output by the multiple parallel deep convolutional branches using learnable weights to obtain multi-scale fused features; subsequently, optimizing the multi-scale fused features through residual structures and activation functions, and restoring the number of channels of the optimized multi-scale fused features to their original state before dimensionality reduction using a linear upprojection layer. Dimensions; Finally, the features after restoring the original dimensions are added to the image features before inputting DMSA through residual connection, and the result is output as the processing result of the current Transformer block; The initial image features are processed sequentially by multiple Transformer blocks with DMSA in the image encoder, and multiple features of different depths are output. These features of different depths together constitute the multi-level image features; Specifically, the layer number of the Transformer block in the image encoder is used as an objective metric, and the layer number comparison rule is set: Among the multiple features of different layers extracted, the features with smaller layer numbers, that is, those ranked before the set relative layer number threshold, are defined as low-level features containing low-level detailed information; the features with larger layer numbers, that is, those ranked after the set relative layer number threshold, are defined as high-level features containing high-level semantic information. In this embodiment, the DMSA is embedded before the first LN in each Transformer block. Given input features , No. i The operation formulas for a Transformer block are as follows: (1); (2); in, MHSA , MLP , LN These represent multi-head self-attention, multilayer perceptron, and layer normalization, respectively. This represents the intermediate transition features obtained after multi-head self-attention and residual connections. Indicates the first i The final output feature of each Transformer block is used as the input feature of the next Transformer block.

[0030] like Figure 2As shown, the calculation process of the Dynamic Multi-Scale Adapter (DMSA) is as follows: First, the input features are processed... The distribution is calibrated by projecting it onto a low-dimensional subspace through a learnable linear layer: (3); in, α and β These are the learnable scaling and offset parameters, respectively. D i Indicates the first i Linear projection layers used for dimensionality reduction in each adapter; Subsequently, three parallel depthwise convolutional branches with kernel sizes of 3×3, 5×5, and 7×7 were used to extract multi-scale features, and learnable weights were applied. w k Dynamic fusion is performed to obtain multi-scale fusion features F. ms : (4); in, DWConv k Indicates the kernel size as k × k deep convolutional layers, It is a minimal constant used for numerical stability.

[0031] Next, lightweight residual blocks are introduced to optimize the multi-scale fusion features: (5); Among them, F ref This represents the optimized multi-scale fusion feature. Represents the GELU activation function. Conv 1×1 This represents a convolutional layer with a kernel size of 1×1; Subsequently, 1×1 convolutions are used to compress and integrate the optimized multi-scale fusion features, and the GELU activation function is used again for non-linear activation. The output is then projected back to the original dimension through another linear layer. Finally, the output of the dynamic multi-scale adapter DMSA is obtained through residual connections. (6); in, This represents the final output feature. U i This indicates a linear projection layer.

[0032] Step 3: Input the multi-level image features into the Multi-Level Feature Cross-Fusion Module (MFCM). Through a cascaded top-down cross-attention mechanism, the high-level semantics and low-level details are fused to obtain the fused features. Specifically, the following steps are taken: the current low-level features are recalibrated using channel attention; cross-attention is calculated using the calibrated current low-level features as the query and the high-level features as the key and value; and the multi-level image features are fused, updated, and stitched together in a top-down cascade manner.

[0033] The SAM encoder extracts features from multiple levels (such as features from layers 3, 6, 9, and 12). , , , To address the issue that simple splicing cannot achieve in-depth interaction, this embodiment uses a top-down cascade fusion based on multi-layer cross-attention MCA.

[0034] like Figure 3 As shown, channel attention is first applied to the input low-level features for recalibration: (7); in, This indicates the current low-level features after channel recalibration. δ Indicates linear projection. GAP This is global average pooling.

[0035] Next, the calibrated low-level features are used as the Query, and the higher-level features are used as the Key and Value for cross-attention calculation: (8); (9); (10); in, Q , K , V These represent the query vector, key vector, and value vector, respectively. Attention ( ) indicates cross-attention calculation. Softmax ( ) represents the Softmax normalized activation function, with the superscript of the parameter. T This indicates that the matrix is ​​transposed. BN Indicates batch normalization. This represents the updated current feature after incorporating high-level semantics.

[0036] The entire calculation process of cross-attention, combining equations (7) to (10), is collectively referred to as multi-layer cross-attention calculation, denoted as the MCA() function. For the deepest feature... Since there is no higher-level semantic reference, only channel attention optimization is performed: (11); The remaining levels are cascaded in a top-down order: (12); Finally, all the refined features are concatenated and processed by convolution to generate fused features. For use by the decoder: (13); in, Concat This indicates that feature splicing is performed along the channel dimension.

[0037] Step 4: Input the fused features into the detail enhancement module DEM, use Discrete Wavelet Transform (DWT) to extract and enhance high-frequency structural details, and generate a prediction mask for salient targets; Specifically, the fused features are input into the mask decoder, and after processing, the mask decoder outputs high-level semantic features. Simultaneously, the mask decoder uses these fused features to directly output an intermediate coarse prediction mask S. m Next, the deepest features from the last layer output of the image encoder are extracted and upsampled to serve as deep encoder features. In the main decoding branch, high-level semantic features are concatenated with deep encoder features and progressively upsampled to restore spatial resolution and obtain the main coarse features. In the auxiliary detail branch, wavelet attention (WA) is introduced, and the local features of the input image are separated into low-frequency sub-bands and multiple high-frequency sub-bands through discrete wavelet transform (DWT). Then, a learnable network parameter threshold is applied to the high-frequency sub-bands (in this embodiment, a soft thresholding function is used to suppress noise in the high-frequency sub-bands). The high-frequency sub-bands after applying the network parameter threshold are then combined with the low-frequency sub-bands, and edge enhancement features are reconstructed through inverse discrete wavelet transform (IDWT) to generate frequency domain enhanced detail features. Finally, the main coarse features and frequency domain enhanced detail features are concatenated and combined to generate the final prediction mask.

[0038] like Figure 4 As shown, in the main decoding branch, the high-level semantic features of the mask decoder are... Features of deep encoders After splicing and compressing the channels, multiple bilinear interpolation upsampling operations are performed to restore spatial resolution. (14); (15); in, This represents the channel compression features obtained by 1×1 convolution dimensionality reduction after concatenation. Indicates the general characteristics of the main trunk. US ×2 This indicates a 2x bilinear interpolation upsampling. Conv 3×3 This indicates a convolutional layer with a kernel size of 3×3.

[0039] In the auxiliary detail branch, wavelet attention WA is used to extract frequency domain details; firstly, local features F of image I are extracted. local : (16); Next, the local features F are transformed using Discrete Wavelet Transform (DWT). local It is decomposed into one low-frequency sub-band L and three high-frequency sub-bands, specifically including vertical... ,level ,diagonal : (17); A learnable threshold is applied to the high-frequency subbands and they are spliced ​​and fused. Then, the low-frequency subbands are combined, and edge enhancement features are reconstructed through inverse discrete wavelet transform (IDWT). (18); in, IDWT ( ) denotes the inverse discrete wavelet transform operation. This represents the edge enhancement features obtained after reconstruction via inverse wavelet transform.

[0040] The wavelet attention weights are generated and multiplied by the original input to obtain the frequency domain enhanced detail features. : (19); in, Flatten ( ) represents the feature flattening operation, which is used to flatten a multidimensional feature tensor into a one-dimensional vector to facilitate the calculation of attention weights.

[0041] Finally, the main trunk coarse features F up With frequency domain enhancement detail features F wa Concatenate the data and output the final prediction mask S. f : (20); Step 5: Train the network model based on the loss function, and use the trained network model to detect salient targets in the image to be detected, and obtain the final prediction mask.

[0042] Step 5 is specifically divided into a model training phase and a model testing phase: During the model training phase: Obtain training image samples with real mask labels, input them into the network model, and process them sequentially through steps 2 to 4; obtain the final output predicted mask S. f and the intermediate coarse prediction mask S output in step 4 m The loss function specifically includes the BCE loss. IoU loss and L1 loss To facilitate gradient propagation in the intermediate layers, the final refined output prediction mask S is prepared separately. f and the intermediate coarse prediction mask S m Calculate the loss above, total loss function for: (twenty one); (twenty two); Where G represents the true mask label of the salient target, and S represents the predicted mask. This represents the loss calculation function between the predicted mask and the true label. This indicates that the final prediction mask S f Calculated loss, This indicates that the intermediate coarse prediction mask S m Calculated loss.

[0043] During the model detection phase: The actual image to be detected is input into the trained network model, and the processing flow of steps 2 to 4 above is executed in sequence. The prediction mask finally output by the DEM is the detection result of the salient target in the image to be detected.

[0044] Example 2: To verify the accuracy and effectiveness of the proposed method for salient object detection in complex scenes, comparative experiments were conducted with other mainstream salient object detection methods on publicly available benchmark datasets (HKU-IS, ECSSD, DUT-OMRON). The quantitative evaluation metrics mainly include mean absolute error (MAE). MAE ), Maximum F-metric ( ), S metric ( ) and average enhanced alignment metric ( E m ).in, MAE The smaller the indicator, the lower the prediction error; , and E m The larger the index, the better the detection accuracy and structural integrity of the model.

[0045] The mainstream methods included in the comparison cover convolutional neural network-based methods (LDF, MENet) and Transformer-based methods (BBRF, ICON-S, SAM, MDSAM). The results of the comparative experiments are shown in Table 1.

[0046] Table 1: Quantitative comparison results with mainstream salient object detection methods on benchmark datasets

[0047] As can be seen from the comparison results in Table 1, compared with traditional methods based on CNN and Transformer, the method of this invention has achieved a significant lead in all indicators, proving the superiority of feature extraction using a visual basic model. In addition, compared with the current advanced similar method MDSAM, the method proposed in this invention also achieves the best performance on three highly representative benchmark datasets.

[0048] Experimental results show that the method proposed in this invention takes into account both global semantics and dynamic adaptation of multi-scale features, and cleverly avoids the loss of details caused by spatial domain decoding through frequency domain processing. Therefore, it can achieve excellent and accurate segmentation of salient targets in challenging scenarios such as lighting changes, multiple targets, small targets or complex edges.

[0049] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0050] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0051] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of the methods disclosed herein and their equivalents, then the intent of this disclosure also includes such modifications and variations.

Claims

1. A dynamic multi-scale salient target detection method based on the SAM large model, characterized in that, Includes the following steps: Step 1: Construct the network model; The network model is based on the SAM architecture and specifically includes an image encoder, a mask decoder, a multi-layer feature cross-fusion module (MFCM), and a detail enhancement module (DEM). Step 2: Input the image to be detected into the image encoder, and use the Dynamic Multi-Scale Adapter (DMSA) to perform dynamic multi-scale feature extraction to obtain multi-level image features; Step 3: Input the multi-level image features into the Multi-Level Feature Cross-Fusion Module (MFCM). Through a cascaded top-down cross-attention mechanism, the high-level semantics and low-level details are fused to obtain the fused features. Step 4: Input the fused features into the detail enhancement module DEM, use Discrete Wavelet Transform (DWT) to extract and enhance high-frequency structural details, and generate a prediction mask for salient targets; Step 5: Train the network model based on the loss function, and use the trained network model to detect salient targets in the image to be detected, and obtain the final prediction mask.

2. The dynamic multi-scale salient target detection method based on the SAM large model according to claim 1, characterized in that, The image encoder described in step 1 is used to receive the image to be detected and extract multi-level image features; the multi-level feature cross-fusion module (MFCM) is connected to the image encoder, receives and fuses the multi-level image features, and outputs the fused features; The mask decoder receives the fused features and outputs high-level semantic features and intermediate coarse prediction masks. The detail enhancement module (DEM) is connected to the image encoder and the mask decoder, respectively. It combines high-level semantic features, deep encoder features, and local features of the input image to generate a predicted mask for the final salient target.

3. The dynamic multi-scale salient target detection method based on the SAM large model according to claim 2, characterized in that, The image encoder consists of several Transformer blocks, each containing a multi-head self-attention (MHSA), a multilayer perceptron (MLP), and a layer normalization (LN). The image encoder embeds a dynamic multi-scale adapter (DMSA). Specifically, the dynamic multi-scale adapter (DMSA) is embedded before the first layer normalization operation of each Transformer block of the image encoder. The Multi-Level Feature Cross-Fusion Module (MFCM) receives multi-level image features output by the image encoder and achieves the fusion of high-level semantics and low-level details through a cascaded cross-attention mechanism, outputting fused features. The detail enhancement module DEM includes a main decoding branch and an auxiliary detail branch; wherein, the main decoding branch is used to receive the high-level semantic features output by the mask decoder and the deep encoder features output by the image encoder, and then concatenate them and upsample them step by step to output the main coarse features; The auxiliary detail branch is used to receive local features of the input image, extract and enhance high-frequency sub-bands through discrete wavelet transform, and output frequency domain enhanced detail features; the main coarse features and the frequency domain enhanced detail features are finally concatenated and combined to generate a prediction mask.

4. The dynamic multi-scale salient target detection method based on the SAM large model according to claim 3, characterized in that, Step 2 specifically involves: inputting the image to be detected into an image encoder, converting the image to be detected into initial image features through block embedding; during the processing of the image encoder, inputting the current image features into a dynamic multi-scale adapter (DMSA), first calibrating and reducing the dimensionality of the current image features using learnable scale and displacement parameters; then, using multiple parallel deep convolutional branches with different receptive field sizes set within the DMSA, extracting multi-scale spatial features from the dimensionality-reduced image features; then, dynamically weighting and fusing the different scale features output by the multiple parallel deep convolutional branches using learnable weights to obtain multi-scale fused features; subsequently, optimizing the multi-scale fused features by sequentially passing them through residual structures and activation functions, and restoring the number of channels of the optimized multi-scale fused features to their pre-dimensionality-reduction state through a linear up-projection layer. The original dimensions are then restored; finally, the features after restoring the original dimensions are added to the image features before inputting DMSA through residual connection, and the result is output as the processing result of the current Transformer block; the initial image features are processed sequentially by multiple Transformer blocks with DMSA in the image encoder, and multiple features of different depths are output. These features of different depths together constitute the multi-level image features; specifically, the layer number of the Transformer block in the image encoder is used as an objective metric, and a layer number comparison rule is set: among the multiple features of different layer numbers extracted, the features with smaller layer numbers, that is, those ranked before the set relative layer number threshold, are defined as low-level features containing low-level detailed information; the features with larger layer numbers, that is, those ranked after the set relative layer number threshold, are defined as high-level features containing high-level semantic information. Given input features , No. i The operation formulas for a Transformer block are as follows: （1）； （2）； in, MHSA , MLP , LN These represent multi-head self-attention, multilayer perceptron, and layer normalization, respectively. This represents the intermediate transition features obtained after multi-head self-attention and residual connections. Indicates the first i The final output features of each Transformer block; The calculation process of the Dynamic Multiscale Adapter (DMSA) is as follows: First, the input features are processed... The distribution is calibrated by projecting it onto a low-dimensional subspace through a learnable linear layer: （3）； in, α and β These are the learnable scaling and offset parameters, respectively. D i Indicates the first i Linear projection layers used for dimensionality reduction in each adapter; Subsequently, three parallel depthwise convolutional branches with kernel sizes of 3×3, 5×5, and 7×7 were used to extract multi-scale features, and learnable weights were applied. w k Dynamic fusion is performed to obtain multi-scale fusion features F. ms : （4）； in, DWConv k Indicates the kernel size as k × k deep convolutional layers, It is a minimal constant used for numerical stability; Next, lightweight residual blocks are introduced to optimize the multi-scale fusion features: （5）； Among them, F ref This represents the optimized multi-scale fusion feature. Represents the GELU activation function. Conv 1×1 This represents a convolutional layer with a kernel size of 1×1; Subsequently, 1×1 convolutions are used to compress and integrate the optimized multi-scale fusion features, and the GELU activation function is used again for non-linear activation. The output is then projected back to the original dimension through another linear layer. Finally, the output of the dynamic multi-scale adapter DMSA is obtained through residual connections. （6）； in, This represents the final output feature. U i This indicates a linear projection layer.

5. The dynamic multi-scale salient target detection method based on the SAM large model according to claim 4, characterized in that, Step 3 specifically involves: recalibrating the current low-level features through channel attention; using the calibrated current low-level features as the query and the high-level features as the key and value, performing cross-attention calculation; and fusing, updating, and stitching the multi-level image features in a top-down cascade manner. Top-down cascaded fusion based on multi-layer cross-attention MCA; First, channel attention is applied to the input low-level features for recalibration: （7）； in, This indicates the current low-level features after channel recalibration. δ Indicates linear projection. GAP For global average pooling; Next, the calibrated low-level features are used as the Query, and the higher-level features are used as the Key and Value for cross-attention calculation: （8）； （9）； （10）； in, Q , K , V These represent the query vector, key vector, and value vector, respectively. Attention ( ) indicates cross-attention calculation. Softmax ( ) represents the Softmax normalized activation function, with the superscript of the parameter. T This indicates that the matrix is ​​transposed. BN Indicates batch normalization. This represents the updated current feature after fusing high-level semantics; The entire calculation process of cross-attention, combining equations (7) to (10), is collectively referred to as multi-layer cross-attention calculation, denoted as the MCA() function; for the deepest feature... Since there is no higher-level semantic reference, only channel attention optimization is performed: （11）； The remaining levels are cascaded in a top-down order: （12）； Finally, all the refined features are concatenated and processed by convolution to generate fused features. For use by the decoder: （13）； in, Concat This indicates that feature splicing is performed along the channel dimension.

6. The dynamic multi-scale salient target detection method based on the SAM large model according to claim 5, characterized in that, Step 4 specifically involves: inputting the fused features into the mask decoder, processing them, and outputting high-level semantic features. Simultaneously, the mask decoder uses these fused features to directly output an intermediate coarse prediction mask S. m Next, the deepest features from the last layer output of the image encoder are extracted and upsampled to serve as deep encoder features. In the main decoding branch, high-level semantic features are concatenated with deep encoder features and progressively upsampled to obtain the backbone coarse features. In the auxiliary detail branch, wavelet attention (WA) is introduced, and the local features of the input image are separated into low-frequency sub-bands and multiple high-frequency sub-bands through discrete wavelet transform (DWT). Then, a learnable network parameter threshold is applied to the high-frequency sub-bands, and the high-frequency sub-bands after applying the network parameter threshold are combined with the low-frequency sub-bands. Edge enhancement features are reconstructed through inverse discrete wavelet transform (IDWT) to generate frequency domain enhanced detail features. Finally, the backbone coarse features and frequency domain enhanced detail features are concatenated and combined to generate the final prediction mask. In the main decoding branch, the high-level semantic features of the mask decoder are... Features of deep encoders After splicing and compressing the channels, multiple bilinear interpolation upsampling operations are performed to restore spatial resolution. （14）； （15）； in, This represents the channel compression features obtained by 1×1 convolution dimensionality reduction after concatenation. Indicates the general characteristics of the main trunk. US ×2 This indicates a 2x bilinear interpolation upsampling. Conv 3×3 This represents a convolutional layer with a kernel size of 3×3; In the auxiliary detail branch, wavelet attention WA is used to extract frequency domain details; firstly, local features F of image I are extracted. local : （16）； Next, the local features F are transformed using Discrete Wavelet Transform (DWT). local It is decomposed into one low-frequency sub-band L and three high-frequency sub-bands, specifically including vertical... ,level ,diagonal : （17）； A learnable threshold is applied to the high-frequency subbands and they are spliced ​​and fused. Then, the low-frequency subbands are combined, and edge enhancement features are reconstructed through inverse discrete wavelet transform (IDWT). （18）； in, IDWT ( ) denotes the inverse discrete wavelet transform operation. This represents the edge enhancement features obtained after reconstruction via inverse wavelet transform; The wavelet attention weights are generated and multiplied by the original input to obtain the frequency domain enhanced detail features. : （19）； in, Flatten ( ) indicates the feature flattening operation; Finally, the main trunk coarse features F up With frequency domain enhancement detail features F wa Concatenate the data and output the final prediction mask S. f : （20）。 7. The dynamic multi-scale salient target detection method based on the SAM large model according to claim 1, characterized in that, Step 5 is specifically divided into a model training phase and a model testing phase: During the model training phase: Obtain training image samples with real mask labels, input them into the network model, and process them sequentially through steps 2 to 4; obtain the final output predicted mask S. f and the intermediate coarse prediction mask S output in step 4 m The loss function specifically includes BCE loss. IoU loss and L1 loss ; respectively in the final refined output prediction mask S f and the intermediate coarse prediction mask S m Calculate the loss above, total loss function for: （21）； （22）； Where G represents the true mask label of the salient target, and S represents the predicted mask. This represents the loss calculation function between the predicted mask and the true label. This indicates that the final prediction mask S f Calculated loss, This indicates that the intermediate coarse prediction mask S m Calculated loss; During the model detection phase: The actual image to be detected is input into the trained network model, and the processing flow of steps 2 to 4 above is executed in sequence. The prediction mask finally output by the DEM is the detection result of the salient target in the image to be detected.

Images