Image salient object detection method based on modality-specific and common feature extraction

Abstract

The present invention discloses an image salient object detection method based on modality-specific and common feature extraction, wherein a modality-common feature extraction module SFA is used to model the strong correlation between two modalities, and a feature erasing module (FEB) is used to model the differences between different modalities; the present invention makes full use of the correlation and differences between multiple modalities, so that the network obtains a more powerful modality fusion and information selection capability; and feature decoding is used to achieve a favorable interaction of various information. The confidence of the salient map of the present invention is higher than that of other advanced methods.

Description

Technical Field

[0001] The present invention belongs to image processing technology, and in particular relates to an image salient object detection method based on modality-specific and common feature extraction. Background Art

[0002] Salient object detection aims to use algorithms to detect and segment areas or objects of interest to the human eye in natural scene images. It is an interdisciplinary scientific problem in the fields of computer science and neurobiology. It can provide useful clues for a variety of visual tasks such as target tracking and pedestrian re-identification, and can greatly improve the results of many visual tasks.

[0003] Early salient object detection methods used traditional methods, that is, using manually extracted color contrast, center priors, and texture features to predict saliency. However, such methods overly rely on manually selected features and perform poorly in some complex situations. In recent years, with the popularization of deep learning, salient object detection technology based on deep learning has developed rapidly, has strong adaptability to scenes, and has achieved better performance than traditional saliency algorithms.

[0004] At present, the unimodal image saliency detection algorithm has achieved a high detection level in existing data sets and simple scenes. However, in complex scenes and multi-interference environments, such as image brightness difference, low color contrast, cross-border and a lot of noise, the effect of unimodal salient object detection needs to be further improved. In order to meet these challenges, in recent years, RGB-T image salient object detection has become a hot spot and difficulty in the field of vision research. RGB-T salient object detection aims to detect salient objects in the scene using information from two modalities, visible light and thermal infrared images. The temperature information provided by thermal infrared helps to improve the salient detection results. Since different modalities are used to predict salient areas, the most important issue in RGB-T salient object detection is to study the correlation and difference between the two modalities.

[0005] Visible light cameras can clearly capture the color and detail information in an image in a well-lit environment, but it is difficult to capture effective information in low light conditions such as at night; thermal infrared cameras are insensitive to changes in lighting and can usually present the outline of the target in the image, but will lose the color and detail information of the image. Therefore, the information provided by visible light images and thermal infrared images is complementary. By extracting the feature information of the two modalities and fusing them, we can obtain richer and more significant target features. At the same time, unique information under different modalities also helps to improve the performance of the algorithm. Therefore, it is necessary not only to achieve the correlation between multiple modalities, but also to consider the differences between modalities.

[0006] Some existing research focuses on the way of information complementation and fusion. Deep learning-based methods predict the final result by fusing multi-scale, multi-modal and multi-level features. These algorithms have made great progress in multi-modal complementarity and achieved better performance. However, these methods only consider the complementarity between multiple modalities and have the following problems:

[0007] On the one hand, due to the different imaging conditions of different modalities, it is unreasonable to simply consider the complementarity of RGB-T and ignore the unique information of each modality;

[0008] On the other hand, studying modal complementarity is an inherent challenge in saliency tasks. Only by establishing common regional features between modalities can we better explore the correlation between modalities. This is not only used to supplement information, but also to distinguish between common features and modality-specific features to better explore the advantages of multimodal salient object detection. Summary of the invention

[0009] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art and to provide an image salient object detection method based on the extraction of modal unique and common features. The present invention refines the common features and unique features of RGB-T multi-modality and utilizes them separately, fully reflecting the advantages of multi-modal salient object detection and obtaining higher performance and more accurate salient images.

[0010] Technical solution: A method for detecting salient objects in an image based on the extraction of modality-specific and common features of the present invention comprises the following steps:

[0011] Step S1: Use a dual-stream feature encoder to encode the multi-level features of the visible light image and the thermal infrared image, and mark the backbone features of the visible light as r 1 ~r 4 , the backbone feature of thermal infrared is marked as t 1 ~t 4 ;

[0012] Step S2: The deepest feature r 4 and t 4 After pyramid pooling PPM processing, we get the global feature G with global receptive field, G = PPM (r 4 ,t 4 );

[0013] Step S3: Use the modal common feature extraction module SFA to extract r 1 ~r 4 and t 1 ~t 4 Perform modal association processing and transfer high-level semantic information to low-level features based on attention, thereby obtaining common regional features F with multimodal common information. F includes F1 ~F 4 ;

[0014] Step S4: Use the feature erasing module FEB to respectively 1 ~r 4 and t 1 ~t 4 Perform modal difference processing to obtain features R and T with unique information of each modality, R includes R 1 ~R 4 and T includes T 1 ~T 4 ;

[0015] Step S5: In the multimodal decoder, first aggregate the difference features to obtain the difference features under multimodality Then, the attention mechanism is used to guide the mutual optimization and fusion of unique information and common information to obtain the salient map S, which includes S 1 ~S 4 ; S 4 is the final saliency map; the unique information and the shared information are aggregated by the multimodal decoder of the present invention;

[0016] Step S6: The training network is supervised by the loss function to obtain the final prediction graph.

[0017] Furthermore, the dual-stream feature encoder in step S1 adopts a Swin-Transformer structure, and the feature map is initially input into a two-layer transformer. The core of the two layers of transformers is ordinary window attention and shift window attention, and then pooling is used between stages to reduce the size of the data to be processed. The number of channels output by each stage is 128, 256, 512 and 1024 respectively.

[0018] Furthermore, the specific method of transmitting the high-level semantic information in step S3 is:

[0019] First, the backbone features of the two modalities are equally divided into four parts, and Sigmoid is used to generate attention masks from high-level features, which are applied to the four features of the two modalities respectively to transfer semantic information between levels. The specific operations are as follows:

[0020]

[0021]

[0022] Among them, Split() is a function integrated by the Pytorch framework, ConvBlock() is a convolution block integrating convolution, normalization and Relu activation functions, σ represents the activation function, Represents high-level features, and Mask represents the weight mask generated by high-level information;

[0023] Then, according to the channel-wise multiplication, Mask is applied to f respectively. r and f t ;

[0024]

[0025] Finally, f′ r and f′ t Channels are merged and the feature context is refined through a convolution block with a convolution kernel size of 3×3. Finally, F is obtained which integrates hierarchical information and context information. r and F t ;

[0026]

[0027] Where [*] is the channel connection operation.

[0028] Furthermore, the modal common feature extraction in step S3 refers to using F r 、F t and G to extract the common regional features. The specific method is:

[0029] In F r 、F t Apply two channel attentions to divide the weight of each channel;

[0030] F′ r =CA(F r )=F r ×σ(AvgPool(F r )+MaxPool(F r ))

[0031] F′ t =CA(F t )=F t ×σ(AvgPool(F t )+MaxPool(F t ))

[0032] Among them, CA() is the channel attention function, AvgPooling and Maxpooling represent average pooling and maximum pooling respectively; then G is processed by a convolution block with an output channel of 128, and finally combined with F′ r and F′t Fusion is performed to extract common area features;

[0033] F=F′ r +F′t+ConvBlock(G)

[0034] Finally, we get the common regional features F of four levels. k , where k = 1, 2, 3, 4.

[0035] Furthermore, after obtaining the common region feature F, it is applied to the two modal features extracted by the encoder to perform an erasing operation, thereby realizing the extraction of modal unique information; the specific operation steps of the modal erasing module are:

[0036] First, the correlation graph of the respective common features and backbone features is calculated at each layer using cosine similarity;

[0037] M (i,j) =(F (i,j) ) T (w T ×f),(1≤i≤H,1≤j≤W)

[0038] Where (i, j) represents the position of each pixel, f represents the backbone feature r or t, w reflects the spatial mapping parameters from f to F, and H is the feature F (i,j) The number of pixels of length, W is the feature F (i,j) The width in pixels;

[0039] Then, a sliding window is used on the correlation graph M to generate a binary mask for the erasure operation, which is defined as

[0040] Finally, using binary mask The common feature F is erased to obtain the features R and T with modality-specific information;

[0041] For example, by using the thermal infrared feature t to erase the common feature F, the visible light unique information feature R will be obtained.

[0042] Furthermore, the specific operation of aggregating the difference features in step S5 is:

[0043] The modality-specific information R and T are merged by channels, and the aggregated unique information is refined using a convolutional block with an output channel of 128 to obtain differential features under multi-modality.

[0044] Furthermore, in step S5, the features of adjacent high layers in F are used to generate an attention mask as the input of multimodal decoding; the difference features, common features and high-level semantic features are aggregated using attention; the specific method of using the attention mechanism is:

[0045] For high-level features First, the low-resolution high-level features are upsampled to the same size as the feature size of the layer, and then the difference features are converted using a convolution block with an output channel of 128. Common features F and high-level features Modify the number of channels to the same, and then use the attention mechanism to automatically obtain the importance of each channel number of the aggregated features;

[0046]

[0047]

[0048] Since there are four layers of features, the four layers of predicted features are obtained, which are respectively passed through a single-channel convolution layer, and then the Sigmoid function is used to constrain the scores to between 0 and 1, which are respectively denoted as S 1 ~S 4 , S 4 The final saliency map

[0049] Furthermore, the details of step S6 are as follows:

[0050] Given a saliency map S = {S(i)|i = {1, ..., N} and true data Y = {Y(i)|i = {1, ..., N}, where N is the total number of pixels in the saliency map;

[0051] The binary cross entropy BCE loss function is:

[0052]

[0053] The smoothness loss function is:

[0054]

[0055]

[0056] The Dice loss function is:

[0057]

[0058] During the training process, the binary cross entropy BCE loss function is used for supervision and applied to the final predicted saliency map;

[0059] L 1 =L(S 4 ,Y);

[0060] The contextual global feature G is then supervised; it is upsampled to the same resolution as Y, and a rough saliency map S is calculated through convolution and activation functions. g ;

[0061] L 2 =L(S g ,Y);

[0062] Finally, smoothness loss and Dice loss are used to supervise the final prediction S;

[0063] L 3 =L s (S 4 ,Y)

[0064] L 4 =L dice (S 4 ,Y)

[0065] Therefore, the final loss function is:

[0066] L final =L 1 +L 2 +L 3 +L 4 .

[0067] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0068] (1) The present invention is based on a modality common and unique feature extraction network, which solves the problem of insufficient utilization of the correlation and difference between modalities in the prior art.

[0069] (2) The feature erasing module FEB of the present invention can effectively realize the refinement of respective modal information, and overcome the problems of unclear texture and inaccurate positioning through the obtained clues.

[0070] (3) The multimodal decoder module of the present invention can fully exploit the advantages of multimodality from the perspective of modal association and difference, thereby obtaining a high-confidence saliency map. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0072] Figure 2 Schematic diagram of a salient object detection network model of the present invention;

[0073] Figure 3 It is a schematic diagram of the PR curve of the present invention;

[0074] Figure 4A visual comparison diagram of the present invention and other technical solutions;

[0075] Figure 5 This is a visualization result diagram of the differential features and common features of the present invention. DETAILED DESCRIPTION

[0076] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0077] like Figure 1 and Figure 2 As shown, the image salient object detection method based on modality-specific and common feature extraction of the present invention comprises the following steps:

[0078] Step S1: Use a dual-stream feature encoder to encode the multi-level features of the visible light image and the thermal infrared image, and mark the backbone features of the visible light as r 1 ~r 4 , the backbone feature of thermal infrared is marked as t 1 ~t 4 ;

[0079] Step S2: The deepest feature r 4 and t 4 After pyramid pooling PPM processing, we get the global feature G with global receptive field, G = PPM (r 4 ,t 4 );

[0080] Step S3: Use the modal common feature extraction module SFA to extract r 1 ~r 4 and t 1 ~t 4 Perform modal association processing and transfer high-level semantic information to low-level features based on attention, thereby obtaining common regional features F with multimodal common information. F includes F 1 ~F 4 ;

[0081] Step S4: Use the feature erasing module FEB to respectively 1 ~r 4 and t 1 ~t 4 Perform modal difference processing to obtain features R and T with unique information of each modality, R includes R 1 ~R 4 and T includes T 1 ~T 4 ;

[0082] Step S5: In the multimodal decoder, first aggregate the difference features to obtain the difference features under multimodality Then, the attention mechanism is used to guide the mutual optimization and fusion of unique information and common information to obtain the salient map S, which includes S 1 ~S 4 ; S 4 is the final saliency map;

[0083] Step S6: The training network is supervised by the loss function to obtain the final prediction graph.

[0084] In this embodiment, the backbone features and global features are associated with each other using a common feature extraction module (SFA), and the specific process of extracting common features in combination with high-level semantic information is as follows:

[0085] First, high-level semantic information is transferred as follows:

[0086] First, the backbone features of the two modalities are equally divided into four parts, and Sigmoid is used to generate attention masks from high-level features, which are applied to the four features of the two modalities respectively to transfer semantic information between levels. The specific operations are as follows:

[0087]

[0088]

[0089] Among them, Split() is a function integrated by the Pytorch framework, ConvBlock() is a convolution block integrating convolution, normalization and Relu activation functions, σ represents the activation function, Represents features from high-level layers, and Mask represents the weight mask generated by high-level information;

[0090] Then, according to the channel-wise multiplication, Mask is applied to f respectively. r and f t ;

[0091]

[0092] Finally, f′ r and f′ t Channels are merged and the feature context is refined through a convolution block with a convolution kernel size of 3×3. Finally, F is obtained which integrates hierarchical information and context information. r and F t ;

[0093]

[0094] Where [*] is the channel connection operation.

[0095] The specific method for extracting the common features of the modalities in step 3 is:

[0096] In F r 、F t Apply two channel attentions to divide the weight of each channel;

[0097] F′ r =CA(F r )=F r ×σ(AvgPool(F r )+MaxPool(F r ))

[0098] F′ t =CA(F t )=F t ×σ(AvgPool(F t )+MaxPool(F t ))

[0099] Among them, CA() is the channel attention function, AvgPooling and Maxpooling represent average pooling and maximum pooling respectively; then G is processed by a convolution block with an output channel of 128, and finally combined with F′ r and F′ t Fusion is performed to extract common area features;

[0100] F=F′ r +F′ t +ConvBlock(G)

[0101] Finally, we get the common regional features F of four levels. k , where k = 1, 2, 3, 4.

[0102] After obtaining the common region feature F, this embodiment applies it to the two modal features extracted by the encoder to perform an erasing operation, thereby realizing the extraction of modal unique information; the specific operation steps of the modal erasing module are:

[0103] First, the correlation graph of the respective common features and backbone features is calculated at each layer using cosine similarity;

[0104] M (i,j) =(F (i,j) ) T (w T ×f),(1≤i≤H,1≤j≤W)

[0105] Where (i, j) represents the position of each pixel, f represents the backbone feature r or t, w reflects the spatial mapping parameters from f to F, and H is the feature F (i,j) The number of pixels of length, W is the feature F (i,j)The width in pixels;

[0106] Then, a sliding window is used on the correlation graph M to generate a binary mask for the erasure operation, which is defined as

[0107] Finally, using binary mask The common feature F is erased to obtain the features R and T with modality-specific information;

[0108]

[0109] The specific operation of aggregating the difference features in step S5 of this embodiment is:

[0110] The modality-specific information R and T are merged by channels, and the aggregated unique information is refined using a convolutional block with an output channel of 128 to obtain differential features under multi-modality.

[0111] Next, the features of adjacent high-level layers in F are used to generate attention masks as the input of multimodal decoding; the difference features, common features, and high-level semantic features are aggregated using attention; the specific method of using the attention mechanism is:

[0112] For high-level features First, the low-resolution high-level features are upsampled to the same size as the feature size of the layer, and then the difference features are converted using a convolution block with an output channel of 128. Common features F and high-level features Modify the number of channels to the same, and then use the attention mechanism to automatically obtain the importance of each channel number of the aggregated features;

[0113]

[0114]

[0115] Since there are four layers of features, the four layers of predicted features are obtained, which are respectively passed through a single-channel convolution layer, and then the Sigmoid function is used to constrain the scores to between 0 and 1, which are respectively denoted as S 1 ~S 4 , S 4 The final saliency map

[0116] The detailed contents of the prediction results obtained in step S6 of this embodiment are:

[0117] Given a saliency map S = {S(i)|i = {1, ..., N} and true data Y = {Y(i)|i = {1, ..., N}, where N is the total number of pixels in the saliency map;

[0118] The binary cross entropy BCE loss function is:

[0119]

[0120] The smoothness loss function is:

[0121]

[0122]

[0123] The Dice loss function is:

[0124]

[0125] During the training process, the binary cross entropy BCE loss function is used for supervision and applied to the final predicted saliency map;

[0126] L 1 =L(S 4 ,Y));

[0127] The contextual global feature G is then supervised; it is upsampled to the same resolution as Y, and a rough saliency map S is calculated through convolution and activation functions. g ;

[0128] L 2 =L(S 4 ,Y);

[0129] Finally, smoothness loss and Dice loss are used to supervise the final prediction S;

[0130] L 3 =L s (S 4 ,Y)

[0131] L 4 =L dice (S 4 ,Y)

[0132] Therefore, the final loss function is:

[0133] L final =L 1 +L 2 +L 3 +L 4 .

[0134] Embodiment 1:

[0135] (1) This embodiment uses three public RGB-T SOD datasets, namely VT5000, VT1000, and VT821.

[0136] Among them, VT5000 is divided into two parts, one is the training set and the other is the test set, which contains 2500 pairs of visible light and thermal infrared images and true values, VT1000 contains 1000 pairs of visible light and thermal infrared images and true values, and VT821 contains 821 pairs of visible light and thermal infrared images and true values.

[0137] (2) In this embodiment, the VT5000 training set is first flipped and cropped to increase the complexity of the training set, and then the processed VT5000 training set is used for training. This embodiment is trained on NVIDIA TITAN Xp GPU for 50 epochs, and the batch size is set to 3. The backbone parameters of the network are the parameters of Swin-B, and the Adams optimizer is used to optimize the network parameters, and the learning rate is 5e -5 , the input image size is 384×384.

[0138] (3) To facilitate quantitative evaluation, this embodiment adopts five widely used indicators.

[0139] (3.1), Mean absolute error (MAE). It represents the average value of the absolute error between the predicted value and the observed value. MAE is defined as:

[0140]

[0141] Where T is the total number of pixels and S is the predicted saliency map, and Y is the true value map.

[0142] (3.2), F-measure (Fm). It is defined as the weighted harmonic mean of precision and recall.

[0143] The F-measure formula is:

[0144]

[0145] where β 2 =0.3, which means more attention is paid to accuracy.

[0146] (3.3), S-measure(S m ). m Calculate the target-aware structural similarity (S) between the predicted image and the true value label 0 ) and region-aware structural similarity (S r ). m As shown below:

[0147] S m =α·S 0 +(1-α)·S r

[0148] Where α is set to 0.5.

[0149] (3.4), E-measure (Em). This measure is an enhanced alignment measure that can measure both global pixel error and local pixel error.

[0150] (3.5) PR curve. It is the precision and recall curve. The precision and recall are calculated using the true value and the predicted graph. The PR curve is drawn by the precision and recall. The closer the PR curve is to the coordinate (1,1), the better the network performance is.

[0151] (4) Compare the technical solution of the present invention with other prior arts.

[0152] This embodiment compares the network of the technical solution of the present invention with other 18 methods.

[0153] The comparison methods include 6 RGB-T SOD methods: MTMR, M3S-NIR, SGDL, ADF, SiamDecoder, SwinNet, 6 single-modal SOD methods: PFA, R3Net, BASNet, PoolNet, CPD, EGNet, and 6 RGBD SOD methods: DMRA, S2MA, UCNet, BBSNet, HAINet, and RD3D. In order to maintain consistency, we retrain the RGBD SOD method using the VT5000 training set.

[0154] All results were generated using code provided by the authors.

[0155] Quantitative comparison:

[0156] This embodiment uses Em, Sm, Fm, MAE and maxF to evaluate the quality of the prediction graph on three data sets, and the specific comparative test results are shown in Table 1. Among all the evaluation indicators, the technical solution of the present invention obtains the highest score compared with other methods.

[0157] Table 1

[0158]

[0159] like Figure 3 As shown, this embodiment gives the PR curve results of the prediction graph and compares them with the results of other technical solutions. It can be observed that the present invention has achieved comparable performance to SwinNet on the three data sets of VT5000, VT1000 and VT821, and has achieved significant improvements compared to other technical solutions. This also illustrates the advantageous results of the technical solution of the present invention.

[0160] Qualitative evaluation:

[0161] like Figure 4 As shown, the comparison results of the visualization of this embodiment and other technical solutions are shown. This implementation selects 8 pairs of relatively challenging samples for display, which can be divided into the following challenging scenarios: small targets (1st and 2nd rows), thermal cross problems (3rd and 4th rows), multi-target problems (5th and 6th rows), and poor quality of thermal infrared images (7th and 8th rows). It can be seen from the results that in the visualization results of some challenging images, the method of the present invention is superior to other comparison methods, and the segmentation effect is significantly improved.

[0162] For the first scenario, small objects in the image are usually difficult to segment and are accompanied by unclear boundaries. In some cases, it can even lead to incorrect predictions. For the second scenario, the imaging quality of the thermal infrared modality is poor, and the generated image is similar to the ambient temperature, and it is impossible to distinguish between the foreground and the background. However, the present invention can filter the background and highlight significant targets through RGB difference information, and segment the targets by effectively utilizing RGB complementary information. In the third scenario, there are multiple significant objects in one picture, resulting in the loss of objects. Other technical solutions have not achieved complete predictions. The results show that the present invention can effectively capture multiple targets. For the last scenario, although poor thermal infrared images will bring a lot of noise, the prediction image of the present invention is still accurate.

[0163] The final visualization results of the visible light and thermal infrared unique features and common features of this embodiment are as follows Figure 5 As shown in Figure 5(a), the thermal infrared feature is shown in Figure 5(a). Figure 5 (b) are common characteristics, Figure 5 (c) is the visible light feature. The thermal infrared of this embodiment tends to focus more on the temperature information in the environment, which can better locate the position of the significant target in the scene, while the visible light tends to focus more on some texture information of the target, which can better refine the edge of the significant target.

[0164] The technical solution of the present invention provides a prediction result with better confidence by combining the difference information and common information of different modes.

Claims

1. A method for detecting salient objects in an image based on the extraction of modality-specific and common features, characterized in that: The following steps are involved: Step S1, using a dual-stream feature encoder to encode and obtain multi-level features of visible light images and thermal infrared images, and marking the backbone features of visible light as r1-r4 and the backbone features of thermal infrared as t1-t4; Step S2, the deepest features r4 and t4 are processed by pyramid pooling PPM to obtain a global feature G with a global receptive field, G = PPM (r4, t4); Step S3, using the modal common feature extraction module SFA to perform modal association processing on r1-r4 and t1-t4, and transfer high-level semantic information to low-level features based on attention, so as to obtain a common regional feature F with multi-modal common information, where F includes F1-F4; The specific method of high-level semantic information transmission is: First, the backbone features of the two modalities are equally divided into four parts, and Sigmoid is used to generate attention masks from high-level features, which are applied to the four features of the two modalities respectively, as follows: Among them, Split() is a function integrated by the Pytorch framework, ConvBlock() is a convolution block integrating convolution, normalization and Relu activation functions, σ represents the activation function, Represents high-level features, and Mask represents the weight mask generated by high-level information; Then, according to the channel-wise multiplication, Mask is applied to f respectively. r and f t ; Finally, f′ r and f′ t Channels are merged and the feature context is refined through a convolution block with a convolution kernel size of 3×3. Finally, F is obtained which integrates hierarchical information and context information. r and F t ; Among them, [*] is the channel connection operation; Using F r 、F t and G to extract the common regional features. The specific method is: In F r 、F t Apply two channel attentions to divide the weight of each channel; F′ r =CA(F r )=F r ×σ(AvgPool(F r )+MaxPool(F r )) F′ t =CA(F t )=F t ×σ(AvgPool(F t )+MaxPool(F t )) Among them, CA() is the channel attention function, AvgPooling and Maxpooling represent average pooling and maximum pooling respectively; then G is processed by a convolution block with an output channel of 128, and finally combined with F′ r and F′ t Fusion is performed to extract common area features; F=F′ r +F′ t +ConvBlock(G) Finally, we get the common regional features F of four levels. k , where k = 1, 2, 3, 4; Step S4, use the feature erasure module FEB to perform modal difference processing on r1-r4 and t1-t4 respectively, so as to obtain features R and T with unique information of their respective modalities, where R includes R1-R4 and T includes T1-T4; the specific operation of the state erasure module is: First, the correlation graph of the respective common features and backbone features is calculated at each layer using cosine similarity; M (i,j) =(F (i,j) ) T (w T ×f),1≤i≤H,1≤j≤W Where (i, j) represents the position of each pixel, f represents the backbone feature r or backbone feature t, w reflects the spatial mapping parameter from f to F, and H is the feature F (i,j) The number of pixels of length, W is the feature F (i,j) The width in pixels; Then, a sliding window is used on the correlation graph M to generate a binary mask for the erasure operation, which is defined as Finally, using binary mask The common feature F is erased to obtain the features R and T with modality-specific information; Step S5: In the multimodal decoder, first aggregate the difference features to obtain the difference features under multimodality Then, the attention mechanism is used to guide the mutual optimization and fusion of unique information and common information to obtain a saliency map S, which includes S1 to S4, and S4 is the final saliency map. The specific method of using the attention mechanism is: For high-level features First, the low-resolution high-level features are upsampled to the same size as the feature size of the layer, and then the difference features are converted using a convolution block with an output channel of 128. Common features F and high-level features Modify the number of channels to the same, and then use the attention mechanism to automatically obtain the importance of each channel number of the aggregated features; Since there are four layers of features, the four layers of predicted features are obtained, which are respectively passed through a single-channel convolution layer, and then the Sigmoid function is used to constrain the scores to between 0 and 1, which are respectively recorded as S1 to S4, and S4 is the final saliency map; Step S6, obtaining the final prediction graph by supervising the training network through the loss function; Given a saliency map S = {S(i)|i = {1, ..., N} and true data Y = {Y(i)|i = {1, ..., N}, where N is the total number of pixels in the saliency map; The binary cross entropy BCE loss function is: The smoothness loss function is: The Dice loss function is: During the training process, the binary cross entropy BCE loss function is used for supervision and applied to the final predicted saliency map; L1=L(S4,Y); The contextual global feature G is then supervised; it is upsampled to the same resolution as Y, and a rough saliency map S is calculated through convolution and activation functions. g ; L2=L(S g ,AND); Finally, smoothness loss and Dice loss are used to supervise the final prediction S; <h2 style=";text-align:left;direction:ltr">L3=L<h2 style=";text-align:left;direction:ltr"> s <h2 style=";text-align:left;direction:ltr"> (S4,Y) L4=L dice (S4,Y) Therefore, the final loss function is: <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> final <h2 style=";text-align:left;direction:ltr"> (L1+L2+L3+L4) 2. The image salient object detection method based on modality-specific and common feature extraction according to claim 1 is characterized in that: The dual-stream feature encoder in step S1 adopts a Swin-Transformer structure. The feature map is initially input into two layers of transformers. The core of these two layers of transformers is ordinary window attention and shift window attention. Then, pooling is used between stages to reduce the size of the data to be processed. The number of channels output by each stage is 128, 256, 512, and 1024, respectively.

3. The image salient object detection method based on modality-specific and common feature extraction according to claim 1 is characterized in that: The specific operation of aggregating the difference features in step S5 is: The modality-specific information R and T are merged by channels, and the aggregated unique information is refined using a convolutional block with an output channel of 128 to obtain differential features under multi-modality.

Images