Method for visible-thermal infrared salient object detection based on correlation modeling

Abstract

The application discloses a visible light-thermal infrared saliency target detection method based on correlation modeling, gradually models the correlation of two modes from the space, region and image level, and predicts accurate saliency detection results in a non-aligned visible light-thermal infrared image pair. At the space level, the correlation modeling is guided to pay attention to the saliency region and align the salient objects in the two modes by embedding an adapter containing salient object semantic information. At the region level, the partial correlation between the two modes is modeled for the aligned salient region. At the image level, the region-level correlation is extended to the whole image of the visible light mode to predict the accurate salient object region in the visible light image. The application achieves the best effect on the non-aligned, weak-aligned and aligned saliency target detection datasets.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and image detection, and particularly relates to a visible light-thermal infrared salient object detection method based on correlation modeling. BACKGROUND

[0002] Salient object detection aims to identify and segment the most attractive regions in a visual scene, which can help to eliminate redundant information and has been applied to various visual tasks, such as image compression, video analysis and visual tracking. Although great progress has been made based on visible light images, existing methods are still difficult to distinguish salient regions in complex scenes, such as cluttered background, low light and similar foreground and background.

[0003] Thermal infrared sensors can capture the overall shape of an object, providing complementary information for visible light images. Therefore, some recent research introduces corresponding thermal infrared images for visible light images for information complementation to improve detection performance in complex scenes. Early traditional visible light-thermal infrared salient object detection methods mainly use hand-crafted features such as texture, color brightness and other information to highlight the salient object regions common in the two modalities by contrast and difference feature fusion. With the improvement of computing power, deep learning-based methods have become mainstream, which train network models using visible light-thermal infrared salient object detection datasets and supervise optimization through corresponding loss functions. The trained model can extract high-level features in visible light and thermal infrared images and fuse them to predict more accurate saliency maps in complex scenes. Deep learning-based methods can be divided into early fusion, mid-level fusion and late fusion according to different fusion methods: early fusion fuses the information of the two modalities before the visible light and thermal infrared image data are sent into the network model, mid-level fusion is performed at the feature level, and late fusion is performed at the prediction result level, among which mid-level fusion has better effect and is the most widely used fusion method.

[0004] However, existing visible light-thermal infrared salient object detection methods are almost designed on the basis of alignment of the two modalities of images. Moreover, the original captured visible light-thermal infrared image pairs are not aligned in space and scale, and manually aligning them will consume a lot of human cost, which is not conducive to practical application and deployment. In addition, due to the difficulty of directly utilizing the multi-modal correspondence in non-aligned image pairs, directly applying existing manual alignment-based methods to non-aligned data will cause a significant decline in performance.

[0005] Existing work, DCNet, attempts to establish the multi-modal correspondence in weakly aligned image pairs through affine transformation and dynamic convolution. The local receptive field of convolution operation is effective in weakly aligned image pairs with small spatial offset, but it is difficult to handle large deviations in space and scale between non-aligned image pairs. In order to improve it, SACNet uses a pair of asymmetric windows to cover the corresponding information in the unaligned image pair. However, the fixed window cannot flexibly realize the accurate coverage of the corresponding information of different scenes, resulting in the introduction of irrelevant noise in the correlation modeling. In the invention, we explicitly align the common information of the two modalities and gradually model the inter-modal and intra-modal correlation.

[0006] Therefore, how to make the network model directly model the correlation of the corresponding information between the two modalities in the non-aligned image pair to utilize the complementary information between the two modalities is the core to improve the performance of the visible light-thermal infrared salient target detection without alignment. SUMMARY

[0007] The purpose of the present application is to solve the problems existing in the prior art and provide a visible light-thermal infrared saliency target detection method based on correlation modeling.

[0008] Technical scheme: The visible light-thermal infrared saliency target detection method based on correlation modeling provided by the present application comprises the following steps:

[0009] Step S1, input a pair of unaligned visible light image and thermal infrared image captured by a camera, the same target captured in the image pair has different spatial positions and scales, then use a pre-trained feature extractor to extract features from the visible light image and the thermal infrared image respectively, and obtain the features of the corresponding modalities, i.e. the multi-layer features of the visible light image and the multi-layer features of the thermal infrared image

[0010] Here, i represents the i-th layer feature, and each modality of the image extracts four layer features, wherein f r 4 and f t 4 are high-level features of the corresponding modalities, which contain rich target semantic information;

[0011] Step S2, input the obtained two high-level features f r 4 and f t 4 into a semantic feature fusion module to integrate the high-level semantic features f s of the common target region in the two modalities, and use the high-level semantic features f s to guide the model to focus on the target region;

[0012] Step S3, the obtained high-level semantic feature f s and the original unaligned visible light image and thermal infrared image are input into a semantic-guided spatial-level correlation module, the spatial-level correlation module models the spatial correlation of the target region in the two modal images, and estimates the homography matrix H between the two modalities on this basis, and the thermal infrared image is warped using the homography matrix H to align the corresponding target region in the visible light image, and the warped thermal infrared image is denoted as I' t ;

[0013] Step S4, the thermal infrared image I' t , the high-level semantic feature f s , the multi-level feature of the visible light image and the multi-level feature of the thermal infrared image are input into a region-level inter-modal correlation module; the region-level inter-modal correlation module first maps the corresponding regions in the visible light modality and the warped thermal infrared modality, and then models the inter-modal correlation of the features of the corresponding regions in the two modalities, and the obtained region-level inter-modal correlation feature is denoted as

[0014] Step S5, the region-level inter-modal correlation feature and the multi-level feature of the visible light image are input into an image-level intra-modal correlation module, the image-level intra-modal correlation module extends the region correlation to the entire image range through self-attention to model the correlation of the salient regions in the entire visible light image, and the obtained intra-modal correlation feature is denoted as

[0015] Step S6, the intra-modal correlation feature is input into a feature decoder to fuse the correlation features of adjacent layers from top to bottom to obtain a predicted saliency map S, which is supervised by the labeled ground truth G.

[0016] To extract discriminative features in the input image, the pre-trained feature extractor in step S1 is a Swin-B model based on Transformer, and the Swin-B model extracts four layers of features with different resolutions, which are 96x96, 48x48, 24x24 and 12x12 respectively.

[0017] To fully utilize the semantic information in the high-level features to locate the target region, step S2 obtains the high-level semantic feature f s through a semantic feature fusion module.

[0018] First, two high-level features f r 4 and f t4 flattening and concatenating to obtain multimodal high-level features

[0019] Then, the multimodal high-level features are sent into self-attention to enhance the target region features that are jointly focused by the two modal features by modeling the dependency in the features;

[0020] Then, a convolutional block is used to refine and smooth the enhanced multimodal high-level features to obtain high-level semantic features f s containing target position information; here, the semantic fusion module includes concatenation, cross-attention, and convolution operations.

[0021] To explicitly align the corresponding regions in the two modalities, the detailed process of step S3 is as follows:

[0022] Step S3.1, using a pre-trained multimodal homography estimator IHN to regard the visible light image and the thermal infrared image as source and target images respectively, and extracting features of the source and target images through a siamese encoder, then calculating the correlation between the source and target images in the feature space and estimating the homography matrix H, the estimation formula of the source and target images is as follows:

[0023]

[0024] where I rgb and I t represent the unaligned visible light image and thermal infrared image respectively, Φ(.) represents a feature encoder, Ψ(,) represents a correlation calculation method, and IHN represents a homography estimator;

[0025] Step S3.2, sending the features extracted by the siamese encoder to a semantic-guided adapter to adapt IHN to visible-thermal infrared data, the calculation process is as follows:

[0026]

[0027] where F is the extracted feature after adaptation of the lth layer and will be propagated to the next layer of features, S-Adapter l is the adapter of the lth layer, S-Adapter l uses the high-level semantic feature f s to guide the current layer feature F l to focus on the salient region, and the calculation process is as follows:

[0028]

[0029] where W dn and Wup It is a mapping operation. φ represents the fusion unit, φ represents the ReLU activation function, CAP(.) represents channel average pooling, σ represents the Sigmoid activation function, and ⊙ represents the element-wise multiplication operation.

[0030] Step S3.3: Adjust the formula for obtaining the homography matrix H by the multimodal homography estimator IHN in step 3.1 as follows:

[0031]

[0032] Where Φ Adapter (,) represents the feature encoder after being fine-tuned by the adapter;

[0033] Step S3.4: Use the homography matrix H obtained in step S3.3 to perform a distortion operation on the thermal infrared image, finally obtaining a thermal infrared image I′ aligned with the visible light. t The calculation process is as follows:

[0034] I' t =Warp(I t ,H)

[0035] Warp(,) is the warp operation.

[0036] To model the correlation between corresponding regions in two modalities, the specific method of step S4 is as follows:

[0037] First, using distorted thermal infrared images I′ t To map the corresponding region in the visible light image, this region is used to interact with the undistorted visible light image. The mapping here refers to IHN taking a pair of source and target images and outputting an estimated common source matrix H. The matrix H can map points in the target image to the corresponding points in the source image.

[0038] Then, the high-level semantic feature f is cited. s To facilitate correlation modeling, focus on the target region and obtain intermodal correlation features. The calculation process is as follows:

[0039]

[0040] in It is the RGB feature of the i-th layer (there are four layers of features). Indicates a mapping operation. Table relevance operations, where Q, K, and V represent query, key, and value, respectively.

[0041] To expand the regional correlation between the two modeled modes, the specific method of step S5 is as follows:

[0042] First, the inter-modal correlation feature is added to the whole visible light modality feature using a residual connection;

[0043] Then, the inter-modal correlation is propagated to the whole visible light modality through a correlation operation to obtain the image-level intra-modal correlation feature The calculation process is as follows:

[0044]

[0045] Wherein, is the RGB feature of the i-th layer, is the region-level inter-modal correlation feature.

[0046] Further, the specific method of the step S6 is:

[0047] The annotated true value G is used to supervise the predicted saliency map S through a binary cross-entropy loss function L bce and a dice coefficient loss function L dice The formula of the total loss function L of the training stage is:

[0048] L=L bce +L dice

[0049]

[0050] Wherein, N is the total number of pixels, G j represents the true value label of the j-th pixel, P j represents the prediction probability of the j-th pixel, |P| represents the number of pixels of the salient target region in the prediction map, |G| represents the number of pixels of the salient target detection region in the true value label map, and |P intersection G| represents the number of pixels of the intersection region between P and G.

[0051] Advantages: compared with the prior art, the present application has the following advantages:

[0052] (1) The present application is based on a progressive multi-modal fusion strategy, and a progressive correlation modeling strategy from the spatial level to the region level and then to the image level is proposed, and the best performance is achieved compared with the prior art.

[0053] (2) The spatial-level correlation modeling module of the present application can fine-tune the pre-trained multi-modal homography estimator to align the corresponding regions in the two modalities.

[0054] (3) The region-level correlation modeling module of the present application can establish the correlation between the features of the corresponding regions of the two modalities.

[0055] (4) The image-level correlation modeling module of the present application can expand the region correlation modeled in the visible light modality to the entire image.

[0056] (5) The semantic feature fusion module of the present application can integrate semantic information in high-level features to help the model locate the target region position. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The figure is a schematic diagram of the overall process of the present application.

[0058] Figure 2 The figure is a schematic diagram of the semantic feature fusion module of the present application.

[0059] Figure 3 The figure is a schematic diagram of the semantic-guided spatial-level correlation module of the present application.

[0060] Figure 4 The figure is a schematic diagram of the region-level inter-modality correlation module of the present application.

[0061] Figure 5 The figure is a schematic diagram of the image-level intra-modality correlation module of the present application.

[0062] Figure 6 The figure is a schematic diagram of the visual comparison between the present application and the prior art. DETAILED DESCRIPTION

[0063] The technical solutions of the present application will be described in detail below, but the protection scope of the present application is not limited to the described embodiments.

[0064] As shown in the figure, the correlation modeling-based visible-thermal infrared saliency target detection method of the present application includes the following steps: Figure 1 Step S1, input a pair of unaligned visible light image and thermal infrared image captured directly by a camera, then use a pre-trained feature extractor to perform feature extraction on the visible light image and the thermal infrared image respectively, to obtain the corresponding modality features, i.e. the multi-layer features of the visible light image

[0065] and the multi-layer features of the thermal infrared image

[0066] Here, i represents the i-th layer feature, and four layers of features are extracted for each modality image, wherein f r 4 and f t 4 are high-level features of the corresponding modality, containing rich target semantic information.

[0067] Step S2, input the two obtained high-level features f r 4 and f t ​4 The data is fed into the semantic feature fusion module, which integrates the high-level semantic features f of the common target region in the two modalities. s Using high-level semantic features f s Guide the model to focus on the target region;

[0068] Step S3: The obtained high-level semantic features f s The original, unaligned visible light image and the thermal infrared image are fed into a semantically guided spatial correlation module. This module models the spatial correlation of the target region in the two modal images and estimates the homography matrix H between the two modalities. The thermal infrared image is then warped using the homography matrix H to align it with the corresponding target region in the visible light image. The warped thermal infrared image is denoted as I′. t ;

[0069] Step S4: Transfer the thermal infrared image I′ t High-level semantic features f s Multi-layer features of visible light images Multilayer features of thermal infrared images Both are fed into the regional-level intermodal correlation module; the regional-level intermodal correlation module first maps the corresponding regions in the visible light mode and the distorted thermal infrared mode, then models the intermodal correlation of the corresponding region features of the two modes, and denotes the resulting regional-level intermodal correlation features as...

[0070] Step S5: Analyze the correlation features between regional modalities. Multi-layer features of visible light images The data is fed into the image-level intramodal correlation module. This module uses self-attention to extend region correlation to the entire image range, modeling the correlation of salient regions within the entire visible light image. The resulting image-level intramodal correlation features are denoted as...

[0071] Step S6: Intramodal correlation features The data is fed into the feature decoder, where the correlation features of adjacent layers are fused from top to bottom to obtain the predicted saliency map S, which is supervised by the labeled ground truth G.

[0072] In step S1 of this embodiment, the pre-trained feature extractor is based on the Transformer Swin-B model. The Swin-B model extracts features from four layers with different resolutions: 96×96, 48×48, 24×24, and 12×12.

[0073] In this embodiment, step S2 obtains high-level semantic features f through the semantic feature fusion module. s The specific method is as follows:

[0074] First, the two high-level features f r 4 and f t 4 Leveling and stitching together yields multimodal high-level features.

[0075] Next, multimodal high-level features By incorporating self-attention, the target region features that are of common interest to both modal features are enhanced by modeling the dependencies in the features.

[0076] Then, a convolutional block is used to refine and smooth the multimodal high-level features to obtain high-level semantic features f containing target location information. s .

[0077] The detailed process of step S3 in this embodiment is as follows:

[0078] Step S3.1: Using a pre-trained multimodal homography estimator (IHN), the visible light image and the thermal infrared image are treated as the source image and target image, respectively. Features of the source image and target image are extracted using a twin encoder. Then, the correlation between the source image and target image is calculated in the feature space, and the homography matrix H is estimated. The estimation formulas for the source image and target image are as follows:

[0079]

[0080] Where I rgb and I t Let represent the misaligned visible light image and thermal infrared image, respectively. Φ(.) represents the feature encoder, and Ψ(,) represents the correlation calculation method. Represents a homography estimator;

[0081] Step S3.2: Input the features extracted from each layer by the twin encoder into the semantically guided adapter, so that the IHN can adapt to visible light-thermal infrared data. The calculation process is as follows:

[0082]

[0083] in The features extracted after the l-th layer adaptation will be propagated to the next layer's features. (S-Adapter) l It is the layer 1 adapter, S-Adapter l Utilizing high-level semantic features f s To guide the current layer features F l Focusing on salient regions, the calculation process is as follows:

[0084]

[0085] Among them W dn and W up It is a mapping operation. φ represents the fusion unit, φ represents the ReLU activation function, CAP(.) represents channel average pooling, σ represents the Sigmoid activation function, and ⊙ represents the element-wise multiplication operation.

[0086] Step S3.3: Adjust the formula for obtaining the homography matrix H by the multimodal homography estimator IHN in step 3.1 as follows:

[0087]

[0088] Where Φ Adapter (,) represents the feature encoder after being fine-tuned by the adapter;

[0089] Step S3.4: Use the homography matrix H obtained in step S3.3 to perform a distortion operation on the thermal infrared image, finally obtaining a thermal infrared image I′ aligned with the visible light. t The calculation process is as follows:

[0090] I' t =Warp(I t ,H)

[0091] Warp(,) is the warp operation.

[0092] The specific method for step S4 in this embodiment is as follows:

[0093] First, using distorted thermal infrared images I′ t To map the corresponding region in the visible light image, this region is used to interact with the undistorted visible light image. Here, mapping means that IHN takes a pair of source and target images and outputs the estimated homology matrix H.

[0094] Then, the high-level semantic feature f is cited. s To facilitate correlation modeling, focus on the target region and obtain regional-level intermodal correlation features. The calculation process is as follows:

[0095]

[0096]

[0097] in It is the RGB feature of the i-th layer. Indicates a mapping operation. Table relevance operations, where Q, K, and V represent query, key, and value, respectively.

[0098] The specific method for step S5 in this embodiment is as follows:

[0099] First, the inter-modal correlation features are added to the whole visible light modal features using residual connection;

[0100] Then, the inter-modal correlation is propagated to the whole visible light modal by correlation operation, obtaining image-level intra-modal correlation features The calculation process is as follows:

[0101]

[0102] wherein, is the RGB feature of the i-th layer, is the region-level inter-modal correlation feature.

[0103] The specific method of step S6 of the embodiment is as follows:

[0104] The annotated ground truth G is used to supervise the predicted saliency map S through the binary cross-entropy loss function L bce and the dice coefficient loss function L dice The total loss function L of the training stage is as follows:

[0105] L = L bce + L dice

[0106]

[0107]

[0108] wherein, N is the total number of pixels, G j represents the ground truth label of the j-th pixel, P j represents the predicted probability of the j-th pixel, |P| represents the number of pixels of the salient target region in the predicted map, |G| represents the number of pixels of the salient target detection region in the ground truth label map, and |P∩G| represents the number of pixels of the intersection region between P and G.

[0109] Embodiment 1:

[0110] For the unaligned model, this embodiment uses 10,000 image pairs from the UVT20K-Train dataset (an existing unaligned visible-thermal infrared salient target detection dataset UVT20K) as the training set, and five unaligned datasets (UVT20K-Test, UVT2000, un-VT5000-Test, un-VT1000, and un-VT821) as the test set to verify the network's effectiveness. For the aligned model, this embodiment uses 2,500 image pairs from the VT5000-Train dataset (an existing aligned visible-thermal infrared salient target detection dataset VT5000) as the training set, and three existing publicly available aligned datasets (VT5000-Test, VT1000, and VT821) as the test set to verify the network's effectiveness.

[0111] like Figure 1 As shown, the visible light-thermal infrared image pairs are respectively fed into a feature extractor to extract the corresponding visible light mode features. and thermal infrared modal characteristics The highest-level feature f r 4 and f t 4 The semantic features are fed into the semantic feature fusion module to obtain semantic features f. s This guides the model to focus on the target region. Simultaneously, the input visible light-thermal infrared image pair and semantic features f are combined. s The corresponding regions in the thermal infrared image and the visible light image are aligned by a semantically guided spatial correlation module to obtain the aligned thermal infrared image I′. t Next, the semantic features, thermal infrared image, visible light modal features, and thermal infrared modal features are fed together into the region-level intermodal correlation module to model the correlation between corresponding region features of two modalities, thus obtaining the intermodal correlation features. Then, the intermodal correlation features and visible light modal features are fed together into the image-level intermodal correlation module to extend the region-level correlation to the entire image level, thereby obtaining the visible light modal correlation features. Finally, the correlation features within the visible light mode are fed into the feature decoder to integrate multi-layer features, resulting in the final predicted saliency map S.

[0112] The network in this embodiment is based on the PyTorch framework and was trained for 80 epochs on two GeForce RTX 3090 GPUs with a batch size of 4. The learning rate was set to 1e-5, the weight decay was 1e-4, and the input image pair resolution was 384*384.

[0113] Three widely used quantitative metrics, E-measure (Em), F-measure (Fm) and S-measure (Sm), are adopted in this example to quantitatively evaluate the performance of the network. The proposed method is compared with other 13 existing methods based on the three quantitative metrics, including MIDD, CSRNet, CGFNet, SwinNet, OSRNet, TNet, DCNet, HRTransNet, MCFNet, LSNet, CAVER, LAFB, and SACNet.

[0114] Quantitative comparison:

[0115] The specific comparison results of this example and the existing 13 technical solutions are shown in Table 1, where the bold font represents the optimal performance. As can be seen from Table 1, compared with the suboptimal method CAVER, the performance of the present application is improved by an average of 6.2%, 2.8% and 7.8% in the three indicators (i.e. Em, Fm and Sm) of the two non-aligned data sets.

[0116] Table 1 Performance comparison results

[0117]

[0118]

[0119] Visual comparison:

[0120] The visual comparison results of the saliency prediction maps generated by this example and the existing 13 technical solutions are shown in Figure 6 This example selects 4 challenging samples including: complex shape of salient object, cluttered image, similar foreground and background of salient object, low light, and low contrast of thermal infrared image. For example, Figure 6 In the first row on the left side, in the case of complex shape of salient object and poor quality of thermal infrared image, the existing advanced methods are difficult to locate the target area or cannot accurately segment the target, while our method can completely and accurately segment the salient object, which proves that our method can fully exploit useful multi-modal complementary information through progressive correlation modeling. For another example, Figure 6 In the second row on the right side, in the case of low light, the visible light image is difficult to provide effective target information, and our method can use the target contour information in the thermal infrared image to achieve accurate prediction, while the compared methods will be seriously disturbed by the background and predict the wrong salient area, which shows that our method can flexibly use useful information in the dominant modality to achieve more accurate prediction. Therefore, the present application can more accurately realize the visible-thermal infrared salient object detection without alignment.

Claims

1. A visible-thermal infrared saliency target detection method based on correlation modeling, characterized in that, Includes the following steps: Step S1: Input a pair of misaligned visible light images and thermal infrared images directly captured by the camera. Then, use a pre-trained feature extractor to extract features from the visible light images and thermal infrared images respectively to obtain the features of the corresponding modes, i.e., the multi-layer features of the visible light images. Multilayer features of thermal infrared images Here, i represents the i-th layer of features. Each modality image has four layers of features extracted, where... and These are high-level features corresponding to the modality, containing rich target semantic information; Step S2: Combine the two high-level features obtained and The data is fed into the semantic feature fusion module, which integrates the high-level semantic features f of the common target region in the two modalities. s Using high-level semantic features f s Guide the model to focus on the target region; Step S3: The obtained high-level semantic features f s The original, unaligned visible light image and the thermal infrared image are fed into a semantically guided spatial correlation module. This module models the spatial correlation of the two modal images for the target region and estimates the homography matrix H between the two modalities. The thermal infrared image is then warped using the homography matrix H to align it with the corresponding target region in the visible light image. The warped thermal infrared image is denoted as I. t ′; Step S4: Transfer the thermal infrared image I t ′、High-level semantic features f s Multi-layer features of visible light images Multilayer features of thermal infrared images Both are fed into the regional-level intermodal correlation module; the regional-level intermodal correlation module first maps the corresponding regions in the visible light mode and the distorted thermal infrared mode, then models the intermodal correlation of the corresponding region features of the two modes, and denotes the resulting regional-level intermodal correlation features as... Step S5: Analyze the correlation features between regional modalities. Multi-layer features of visible light images The data is fed into the image-level intramodal correlation module. This module uses self-attention to extend region correlation to the entire image range, modeling the correlation of salient regions within the entire visible light image. The resulting image-level intramodal correlation features are denoted as... Step S6: Intramodal correlation features The data is fed into the feature decoder, where the correlation features of adjacent layers are fused from top to bottom to obtain the predicted saliency map S, which is supervised by the labeled ground truth G.

2. The visible-thermal-infrared saliency target detection method based on correlation modeling according to claim 1, characterized in that, The pre-trained feature extractor in step S1 is based on the Transformer Swin-B model. The Swin-B model extracts features from four layers with different resolutions: 96×96, 48×48, 24×24, and 12×12.

3. The visible-thermal-infrared saliency target detection method based on correlation modeling according to claim 1, characterized in that, Step S2 obtains high-level semantic features f through the semantic feature fusion module. s The specific method is as follows: First, the two high-level features and Leveling and stitching together yields multimodal high-level features. Next, multimodal high-level features By incorporating self-attention, the target region features that are of common interest to both modal features are enhanced by modeling the dependencies in the features. Then, a convolutional block is used to refine and smooth the multimodal high-level features to obtain high-level semantic features f containing target location information. s .

4. The visible-thermal infrared saliency target detection method based on correlation modeling according to claim 1, characterized in that, The detailed process of step S3 is as follows: Step S3.1: Using a pre-trained multimodal homography estimator (IHN), the visible light image and the thermal infrared image are treated as the source image and target image, respectively. Features of the source image and target image are extracted using a twin encoder. Then, the correlation between the source image and target image is calculated in the feature space, and the homography matrix H is estimated. The estimation formulas for the source image and target image are as follows: Among them I rgb and I t Let represent the misaligned visible light image and thermal infrared image, respectively. Φ(.) represents the feature encoder, and Ψ(,) represents the correlation calculation method. Represents a homography estimator; Step S3.2: Input the features extracted from each layer by the twin encoder into the semantically guided adapter, so that the IHN can adapt to visible light-thermal infrared data. The calculation process is as follows: in The features extracted after the l-th layer adaptation will be propagated to the next layer's features. (S-Adapter) l It is the layer 1 adapter, S-Adapter l Utilizing high-level semantic features f s To guide the current layer features F l Focusing on salient regions, the calculation process is as follows: Among them W dn and W up It is a mapping operation. φ represents the fusion unit, φ represents the ReLU activation function, CAP(.) represents channel average pooling, σ represents the Sigmoid activation function, and ⊙ represents the element-wise multiplication operation. Step S3.3: Adjust the formula for obtaining the homography matrix H by the multimodal homography estimator IHN in step 3.1 as follows: Where Φ Adapter (,) represents the feature encoder after being fine-tuned by the adapter; Step S3.4: Use the homography matrix H obtained in step S3.3 to perform a distortion operation on the thermal infrared image, finally obtaining a thermal infrared image I aligned with the visible light. t The calculation process is as follows: I t ′=Warp(I t ,H) Warp(,) is the warp operation.

5. The visible-thermal infrared saliency target detection method based on correlation modeling according to claim 1, characterized in that, The specific method for step S4 is as follows: First, using distorted thermal infrared images I t The corresponding region in the visible light image is mapped to the undistorted visible light image. This region is used to interact with the undistorted visible light image. Here, mapping means that IHN takes a pair of source and target images and outputs the estimated homogeneous matrix H. Then, the high-level semantic feature f is cited. s To facilitate correlation modeling, focus on the target region and obtain regional-level intermodal correlation features. The calculation process is as follows: in It is the RGB feature of the i-th layer. Indicates a mapping operation. Table relevance operations, where Q, K, and V represent query, key, and value, respectively.

6. The visible-thermal infrared saliency target detection method based on correlation modeling according to claim 1, characterized in that, The specific method for step S5 is as follows: First, residual connections are used to add the intermodal correlation features to the entire visible light modal features; Then, the correlation between modes is propagated to the entire visible light mode through correlation operations to obtain image-level intramodal correlation features. The calculation process is as follows: in, It is the RGB feature of the i-th layer. It is a feature of intermodal correlation at the regional level.

7. The visible-thermal infrared saliency target detection method based on correlation modeling according to claim 1, characterized in that, The specific method for step S6 is as follows: Using the labeled truth value G, the binary cross-entropy loss function L... bce and dice coefficient loss function L dice The formula for the saliency map S of the jointly supervised prediction and the total loss function L during the training phase is: L=L bce +L dice Where N is the total number of pixels, G j Let P be the truth label of the j-th pixel. j Let |P| represent the predicted probability of the j-th pixel, |P| represent the number of pixels in the salient target region in the predicted image, |G| represent the number of pixels in the salient target detection region in the ground truth label image, and |P∩G| represent the number of pixels in the intersection region between P and G.

Images