Multimodal salient object detection method based on dynamic routing combined with asymmetric multi-window

Abstract

The present invention provides a multi-modal salient target detection method combining dynamic routing with asymmetric multi-windows, belonging to the field of image processing. The steps are as follows: S1: Use two parallel neural network branches to process visible light and thermal infrared image pairs to form visible light and thermal infrared feature maps of each layer; S2: Use different size windows to divide and obtain large feature maps and small feature maps of different layers; S3: Apply dynamic path selection strategy to obtain high-level features after weighted aggregation, obtain global features and obtain enhanced features of different layers; S4: Obtain aligned features of different layers; S5: Combine the zeroth layer of visible light and thermal infrared feature maps to obtain predicted saliency maps. The present invention dynamically selects and weights features of different layers, and uses visual attention mechanism to perform multi-size window modeling for different modal correlations of unaligned salient areas, which overcomes the problem that data sets need to be aligned in RGBT salient target detection, and has excellent prediction performance.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a multi-modal salient target detection method combining dynamic routing with asymmetric multi-windows. Background Art

[0002] As a key technology in the field of computer vision, salient object detection aims to quickly and accurately identify the most visually prominent objects in complex scenes. This method simulates the attention mechanism of the human visual system and gives higher priority to salient objects in the image, thereby providing strong support for subsequent image understanding and analysis. Salient object detection technology has a wide range of application value in real life. It can not only be used in image segmentation, video surveillance and other fields, but also provide core technical support for image retrieval, target tracking and autonomous driving. Salient object detection in visible light images will face challenges in environments with poor lighting conditions. Since thermal infrared images are based on the heat emitted by the object itself, they can still clearly show the thermal distribution of the target at night or in dimly lit environments. They have unique advantages in poor lighting conditions and can effectively make up for the limitations of visible light images in low light conditions.

[0003] The main purpose of salient object detection based on visible light modality and thermal infrared modality is to improve the detection ability of salient objects in complex environments and provide more stable and reliable results for various visual tasks. RGBT salient object detection achieves efficient salient object detection in complex environments by fusing the features of the two modalities and cleverly utilizing the complementary information of the two modalities of visible light and thermal infrared. This method combines the rich texture information of visible light images and the stable thermal radiation information of thermal infrared images, making the detection results have achieved remarkable results in terms of accuracy and robustness. However, most of the current research on this method is based on manually aligned datasets for experiments. Its limitation is that it is difficult to generalize to the actual collected datasets that have not been manually aligned. In addition, the manual alignment process is not only time-consuming and labor-intensive, but also limits its wide promotion in practical applications. Since the manually aligned salient object image pairs are consistent in space and size, simple addition or concatenation operations can obtain better fusion effects. However, when these methods are applied to unaligned original datasets, due to the lack of accurate alignment information, their performance often drops significantly, which affects the detection results.

[0004] The paper "Weakly alignment-free RGBT salient object detection with deep correlation network" (Zhengzheng Tu, Zhun Li, Chenglong Li, and Jin Tang. IEEE Trans. Image Process., vol. 31, pp. 3752–3764, 2022) explores salient object detection for weakly aligned datasets in RGBT mode, using spatial affine transformation and dynamic convolution to model the correlation between the two modalities. However, since weakly aligned datasets are less challenging than the original collected datasets, and the affine transformation space is limited and the receptive field of dynamic convolution is small, the model's adaptability to complex scenes is poor. The paper "Alignment-Free RGBT Salient Object Detection: Semantics-Guided AsymmetricCorrelation Network and a Unified Benchmark" (Kunpeng Wang, Danying Lin, Chenglong Li, Zhengzheng Tu, Bin Luo. inIEEE Transactions on Multimedia, doi:10.1109 / TMM.2024.3410542) constructs a benchmark dataset UVT2000 without manual alignment, which contains 2000 pairs of RGB and thermal infrared images directly obtained from various real scenes, and proposes a semantically guided asymmetric association module to model the cross-modal correlation of non-aligned salient areas. However, there are still two problems: (1) Alignment-Free directly uses a certain high-level feature for semantic guidance, which ignores the fact that challenging inputs in actual complex scene tasks will cause different high-level features to contribute differently to each possible salient target area; (2) Alignment-Free uses semantically guided attention to model the cross-modal correlation of misaligned salient regions with fixed-size windows, ignoring the contribution of multi-size window modeling to the cross-modal correlation of misaligned salient regions. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to break the limitations of the static single-layer architecture and overcome the dataset alignment problem in RGBT salient object detection, and improve the quality of cross-modal feature correspondence.

[0006] In order to solve the above technical problems, the present invention provides the following technical solution: a multi-modal salient target detection method combining dynamic routing with asymmetric multi-windows, comprising the following steps:

[0007] S1: Obtain image pairs under different modes, i.e., visible light image and thermal infrared image pairs, input them into the neural network, and use two parallel neural network branches to process the input image pairs to form visible light feature maps of each layer. Thermal infrared signature , i Representation level;

[0008] S2: Yes , Use windows of different sizes to divide and obtain large feature maps of different layers of feature maps under two modes , With small feature map , ;

[0009] S3: Dynamically weight the features at different levels under the two modes and then perform feature operations to obtain global features and , combined with , , semantically guide the features of different layers under the two modalities to obtain the enhanced features of different layers and ;

[0010] S4: According to and , combined with and Get the aligned features of different layers and ;

[0011] S5: Based on the aligned features of different layers and The predicted saliency map is obtained by combining the visible light feature map and the thermal infrared feature map at the zeroth layer.

[0012] Preferably, the neural network in step S1 is Swin Transformer.

[0013] The present invention uses Swin Transformer as the backbone network, which adopts a windowed attention mechanism to divide the image into non-overlapping windows, apply self-attention in each window, and adopts a shift window technology. In consecutive attention layers, the position of the window will shift, allowing information between different windows to interact.

[0014] Preferably, in step S1, , that is, both the visible light feature map and the thermal infrared feature map have 5 layers, and the feature maps of each layer of visible light are marked as ~ , the characteristic maps of each layer of thermal infrared are marked as ~ .

[0015] Preferably, the specific process of step S2 is:

[0016] S201: The feature maps with the same number of layers in two modes are taken as a pair of feature maps;

[0017] S202: For the three pairs of feature maps corresponding to the third layer, the second layer, and the first layer under the two modes: and , and , and Use three pairs of windows of different sizes for division;

[0018] S203: Obtaining the size feature map of the visible light mode , and the size characteristic diagram of thermal infrared mode , , i= 1, 2, 3.

[0019] Preferably, in step S202, the window sizes of the visible light feature map and the thermal infrared feature map of the same layer are the same.

[0020] Due to the first-level characteristics and High resolution, high computational effort, and the highest level features and Because a lot of detail information is lost after multiple downsampling and abstraction, the present invention only performs window division on the feature maps of the third layer, the second layer, and the first layer in the two modalities. The multi-size window division strategy of the present invention allows the model to capture detail information at a fine-grained level while maintaining an understanding of the overall structure at a coarser scale. The model can establish correspondence between features in different modalities at different scales, thereby improving the modeling effect of cross-modal correlations in non-aligned salient areas.

[0021] Preferably, the specific process of step S3 is:

[0022] S301: Obtain routing probabilities of the second, third, and fourth layer feature maps of the visible light modality and the thermal infrared modality;

[0023] S302: Assign the contribution weight of each layer of features in the high-level semantics according to the routing probability value of each layer of features, and obtain the high-level features after weighted aggregation of the two modalities , ;

[0024] S303: High-level features after weighted aggregation of the two modalities , Flatten and concatenate to obtain ,right Perform correlation operations to obtain the features after multimodal fusion ;

[0025] S304: Features after multi-modal fusion Decomposed into two modal global features in spatial dimension and ,right and Perform upsampling and convolution operations to achieve size adaptation;

[0026] S305: After resizing and Upsampling, and , and After multiplication, convolution operation is performed to obtain enhanced features of different layers. and , i= 1,2,3.

[0027] Due to the first-level characteristics and The computation is large, and the highest level features and The resolution is small and the features are too abstract, so the present invention only performs semantic guidance on the feature maps of the middle three layers under the two modalities, that is, semantic guidance is performed on the third layer, the second layer, and the first layer.

[0028] Preferably, the specific process of step S301 is:

[0029] S301-1: The second layer feature map of the visible light mode and thermal infrared mode and , the third layer feature map and , the fourth layer feature map and As three inputs to the routing module;

[0030] S301-2: Yes and conduct Up Upsampling to get ' and ' ;

[0031] S301-3: Yes ' and ' , and , and The feature maps obtained after average pooling and maximum pooling are concatenated in the feature dimension , , ;

[0032] S301-4: Yes , , Flattening operation is performed to obtain ' , , ;

[0033] S301-5: Yes ' , , Perform concatenation, then apply MLP and use The function obtains the routing probabilities of the second, third, and fourth layer features under the two modes.

[0034] Step S301 of the present invention adopts a dynamic routing selection mechanism, which can help the neural network better understand the relationship between features in different layers, and selectively strengthen the features that have important contributions to non-aligned salient targets, and then output aggregated feature representations that better encode the salient features in the input image.

[0035] Preferably, in step S4, the , , and In i =1,2,3, the specific process of step S4 is:

[0036] S401: Enhance the features of the first, second and third layers respectively and Flattening operation is performed to obtain and ;

[0037] S402: Yes and , and Perform correlation operations to obtain aligned features of different layers and .

[0038] Preferably, in step S5, the aligned features of different layers and In i =1,2,3, the specific process of step S5 is:

[0039] S501: and Perform deformable convolution to obtain fused features , and Perform deformable convolution to obtain , and Perform deformable convolution to obtain ;

[0040] S502: According to , , Get the saliency map of the preliminary prediction;

[0041] S503: using the visible light zeroth layer feature map and the thermal infrared zeroth layer feature map, supplementing the preliminary predicted saliency map with detailed information;

[0042] S504: Obtain the final predicted saliency map.

[0043] Preferably, the specific process of step S502 is:

[0044] S502-1: Yes After upsampling, merge;

[0045] S502-2: Upsample the merged result, and merge;

[0046] S502-3: Upsample the merged result at this time to obtain a preliminary predicted saliency map.

[0047] Compared with the prior art, the advantages of the present invention are:

[0048] (1) The multi-size window strategy is applied to divide the feature maps of different layers into windows of different sizes, which enhances the prediction model's ability to capture features of different sizes. It allows the model to capture detailed information at a fine-grained level while maintaining an understanding of the overall structure at a coarser scale. It can establish the correspondence between features in different modalities at different scales, align the salient information in the two modalities as much as possible, and improve the modeling effect of cross-modal correlation of non-aligned salient regions. It overcomes the problem of dataset alignment in RGBT salient object detection and can improve the generalization ability of modal fusion methods in practical applications.

[0049] (2) A dynamic routing selection mechanism is applied to dynamically select useful layers of features based on the semantic information of the high-level features, which are the second, third, and fourth layers in this invention, and weight them, which can help the neural network better understand the relationship between features of different layers and selectively strengthen features that contribute significantly to non-aligned salient targets. The aggregated feature representation is then output, which can better encode the salient features in the input image, improve the performance of the neural network and the accuracy of prediction, and break the limitation of the traditional attention scheme that adopts a static single-layer architecture.

[0050] (3) Considering the characteristics of the first layer and High resolution, large amount of calculation, highest level features and The resolution is low and the features are too abstract, so the present invention only uses three pairs of windows of different sizes to perform window division and semantic guidance on the feature maps of the first, second and third layers, which can reduce the amount of calculation and improve the prediction efficiency. In addition, the first-layer features are used to supplement the detailed information when the prediction map is finally output, thereby ensuring the accuracy of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is an overall framework diagram of the embodiment of the present invention;

[0052] Figure 2 It is a process framework diagram of the dynamic routing selection mechanism in an embodiment of the present invention;

[0053] Figure 3 A quantitative evaluation diagram for evaluating the performance of the method of the present invention;

[0054] Figure 4 It is a qualitative evaluation diagram for evaluating the performance of the method of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Example

[0057] like Figure 1 As shown, this embodiment provides a multi-modal salient target detection method combining dynamic routing with asymmetric multi-windows, comprising the following steps:

[0058] Step 1: Obtain image pairs under different modes, namely visible light images and thermal infrared images, input them into the SwinTransformer neural network, and use two parallel neural network branches to process the input image pairs to form visible light feature maps of each layer. Thermal infrared signature , i Represents a hierarchy.

[0059] The visible light characteristic map and the thermal infrared characteristic map in this embodiment both have 5 layers, namely . The feature maps of each layer of visible light are marked as ~ , the characteristic maps of each layer of thermal infrared are marked as ~ .

[0060] Step 2: Since the first-layer features have high resolution and large computational complexity, and the highest-layer features have lost a lot of detail information due to multiple downsampling and abstraction, this embodiment only divides the third-layer, second-layer, and first-layer feature maps under the two modalities using windows of different sizes to obtain large feature maps of the third-layer, second-layer, and first-layer feature maps under the two modalities. , With small feature map , , i= 1, 2, 3, the specific process is:

[0061] S201: The feature maps with the same number of layers in two modes are taken as a pair of feature maps;

[0062] S202: For the three pairs of feature maps corresponding to the third layer, the second layer, and the first layer under the two modes: and , and , and Three pairs of large and small windows of different sizes are used for division. The large and small windows for dividing the visible light feature map and the thermal infrared feature map of the same layer are of the same size. The large window size of each pair of windows is represented by M, and the corresponding small window size is represented by N.

[0063] For example, for the third layer feature map and Use a large window of M=6 and a small window of N=4 to divide the second layer feature map and Use a large window of M=6 and a small window of N=2 to divide the first layer feature map and Use a large window of M=4 and a small window of N=2 for division. The sizes of M and N are not limited to the following;

[0064] S203: Obtaining large feature maps of the third, second, and first layers of the visible light modality With small feature map And the large feature maps of the third, second and first layers of the thermal infrared modality With small feature map , i= 1, 2, 3.

[0065] Step 3: Apply the dynamic routing selection mechanism to weight the features of the second, third, and fourth layers of the two modes, and then perform feature operations to obtain global features. and , combining large feature maps of different layers under two modalities , , semantically guide the features of different layers under the two modalities to obtain the enhanced features of different layers and , the specific process is:

[0066] S301: Figure 2 As shown, the routing probabilities of the second, third, and fourth layer feature maps of the visible light modality and thermal infrared modality are obtained. The specific process is:

[0067] S301-1: The second layer feature map of the visible light mode and thermal infrared mode and , the third layer feature map and , the fourth layer feature map and As three inputs to the routing module;

[0068] S301-2: Yes and conduct Up Upsampling to get ' and ' ;

[0069] S301-3: For two modes 'and '、 and , and The feature maps obtained after average pooling and maximum pooling are concatenated in the feature dimension , , , the formula is:

[0070] , where , avgpoo l represents the average pooling operation, maxpool represents the maximum pooling operation, concat Represents a splicing operation;

[0071] S301-4: Yes '、 , Flattening operation is performed to obtain '、 , , the formula is: , where flatten Represents a flattening operation;

[0072] S301-5: Yes '、 , After flattening, apply MLP and use The function obtains the routing probability of the second, third, and fourth layer features under the two modes, and the formula is:

[0073] , where i =1,2,3, where w1 is the routing probability of the second layer feature, w2 is the routing probability of the third layer feature, w3 is the routing probability of the fourth layer feature;

[0074] S302: Assign the contribution weight of each layer of features in the high-level semantics according to the routing probability value of each layer of features, and obtain the high-level features after weighted aggregation of the two modalities , ;

[0075] S303: Since the high-level features of the two modalities themselves lack modal complementarity, which affects the detection of salient regions shared by the two modalities, the high-level features after weighted aggregation of the two modalities are , Flatten and concatenate to obtain , the formula is: , where represents the flattening operation, Concate Indicates a series operation, and then Perform correlation operations to obtain the features after multimodal fusion , the formula is: , where correlation Represents correlation operations;

[0076] S304: Features after multi-modal fusion Decomposed into two modal global features in spatial dimension and ,right and Perform upsampling and convolution operations to achieve size adaptation;

[0077] S305: After resizing and Upsampling, with , After multiplication, convolution operation is performed to obtain enhanced features of different layers. and , the formula is: ,

[0078] , where ⊙ represents point-by-point multiplication, Up x Indicates that bilinear interpolation is used x times upsampling, Conv represents the convolution operation, i = 1,2,3.

[0079] Step 4: Enhanced features according to different layers and , combined with and Get the aligned features of different layers and , i =1,2,3, the specific process is:

[0080] S401: Enhance the features of the first, second and third layers respectively and Flattening operation is performed to obtain and , the formula is: , , where flatten Represents a flattening operation;

[0081] S402: and Respectively and Perform correlation operations to obtain aligned features of different layers and , the formula is: , , where Correlation Represents a dependency operation.

[0082] Step 5: Based on the aligned features of different layers and The predicted saliency map is obtained by combining the visible light feature map and the thermal infrared feature map at the zeroth layer. The specific process is as follows:

[0083] S501: and Perform deformable convolution to obtain fused features , and Perform deformable convolution to obtain , and Perform deformable convolution to obtain ;

[0084] S502: After upsampling Merge, upsample the merged result and Merge, and upsample the merged result again to obtain the preliminary predicted saliency map;

[0085] S503: Based on the high computational cost of the first-layer high-resolution feature map, only the visible light zero-layer feature map is used here Thermal infrared layer zero feature map Supplement detailed information to the preliminary predicted saliency map;

[0086] S504: Obtaining the final predicted saliency map P .

[0087] In this embodiment, a hybrid function is used to optimize the multimodal salient target detection method, and the hybrid function formula is expressed as: ,in express cross-entropy Loss function, express smoothness Loss function, express dice Loss function, P is the predicted saliency map, GT is the true label.

[0088] To verify the effectiveness of the method of this embodiment, this embodiment conducts experiments on 7 RGBT salient object detection datasets, including three aligned datasets un-VT821, un-VT1000, un-VT5000 and three weakly aligned datasets UVT821, UVT1000, UVT5000 and one non-aligned dataset UVT2000. The three weakly aligned datasets are obtained by performing weak alignment transformation operations on the three datasets VT821, VT1000, and VT5000 respectively.

[0089] VT821 provides a dataset of 821 pairs of spatially aligned RGB-T images and their real annotations for salient detection, collected by online thermal imagers and CCD cameras. These image pairs are collected from 60 scenes under different environmental conditions and have a high degree of diversity. VT1000 is a salient object detection dataset containing 1,000 pairs of RGBT image pairs, which is a further improvement on the problems of VT821. VT5000 creates an RGBT image dataset containing 5,000 pairs of RGB and thermal infrared images, providing a benchmark method to evaluate the performance of different methods in the RGBT salient object detection task. VT5000 has the characteristics of large scale, high resolution, high diversity, and low deviation, taking into account the different sizes, categories, surrounding environments, number of imaging, and spatial positions of salient objects. The UVT2000 dataset contains 2,000 pairs of RGB and thermal infrared imaging combinations that have not been manually aligned.

[0090] In order to facilitate the comparison of the performance of different RGBT salient target detection models, this embodiment uses the indicators commonly used in salient target detection for evaluation: F-measure (Fβ), S-measure (Sα), E-measure (Em), and MAE (M). The higher the values ​​of Fβ, Sα, and Em, the better, and the lower the value of MAE, the better. Fβ is the harmonic mean of precision and recall. Under normal circumstances, neither precision nor recall can fully evaluate the quality of saliency prediction. Therefore, Fβ is chosen to make a comprehensive judgment on accuracy and completeness. Sα is developed from F-measure and aims to comprehensively evaluate the accuracy and structural similarity of saliency maps. Em is used to measure the degree of alignment between the predicted saliency map and the true saliency map. MAE is used to evaluate the average absolute difference between the generated saliency map and the true saliency annotation.

[0091] like Figure 3The figure shows the experimental results of the method of this embodiment on various salient object detection datasets. By comparison, it can be seen that the performance indicators of the saliency map prediction using the method of this embodiment are improved compared with the traditional method, especially the performance improvement on the non-aligned dataset UVT2000 is obvious, that is, the method of the present invention can overcome the dataset alignment problem in RGBT salient object detection.

[0092] like Figure 4 As shown in the figure, this embodiment is experimented on UVT2000, a large-scale, unaligned real-scene salient object detection dataset. The columns labeled "Ours" in the figure show the experimental visualization results of the present invention, while the columns labeled "SACNet" show the visualization effect of the baseline method that is the basis for the improvement of the present invention, that is, the present invention has achieved innovation and improvement on the basis of the baseline method. In addition, the columns labeled "SwinNet" represent the experimental performance of the currently popular methods. By comparison, it can be clearly observed that the method of the present invention has significant advantages over other technologies.

[0093] This invention breaks the limitation of the static single-layer architecture of traditional attention schemes, and proposes an asymmetric multi-window semantic correlation network guided by a new dynamic routing algorithm for salient object detection in unaligned images. First, a dynamic path selection strategy is used to dynamically select and weight features of different layers according to the semantic information of high-level features. Secondly, a semantically guided visual attention mechanism (Transformer) is used to model multi-size windows for different modal correlations in unaligned salient regions. Experiments are conducted on large-scale salient object detection datasets from various real scenes without any alignment, verifying the effectiveness and superior performance of the invention.

[0094] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal salient object detection method combining dynamic routing with asymmetric multi-windows, characterized in that: The following steps are involved: S1: Obtain image pairs under different modes, i.e., visible light image and thermal infrared image pairs, input them into the neural network, and use two parallel neural network branches to process the input image pairs to form visible light feature maps of each layer. Thermal infrared signature , i Representation level; S2: Yes , Use windows of different sizes to divide and obtain large feature maps of different layers of feature maps under two modes , With small feature map , ; S3: Dynamically weight the features at different levels under the two modes and then perform feature operations to obtain global features and , combined with , , semantically guide the features of different layers under the two modalities to obtain the enhanced features of different layers and , specifically: S301: Obtain routing probabilities of the second, third, and fourth layer feature maps of the visible light modality and the thermal infrared modality; S302: Assign the contribution weight of each layer of features in the high-level semantics according to the routing probability value of each layer of features, and obtain the high-level features after weighted aggregation of the two modalities , ; S303: High-level features after weighted aggregation of the two modalities , Flatten and concatenate to obtain ,right Perform correlation operations to obtain the features after multimodal fusion ; S304: Features after multi-modal fusion Decomposed into two modal global features in spatial dimension and ,right and Perform upsampling and convolution operations to achieve size adaptation; S305: After resizing and Upsampling, and , and After multiplication, convolution operation is performed to obtain enhanced features of different layers. and , i =1,2,3; S4: According to and , combined with and Get the aligned features of different layers and ; S5: Based on the aligned features of different layers and The predicted saliency map is obtained by combining the visible light feature map and the thermal infrared feature map at the zeroth layer.

2. According to claim 1, a multi-modal salient object detection method combining dynamic routing with asymmetric multi-windows is characterized in that: The neural network in step S1 is Swin Transformer.

3. According to claim 1, a multi-modal salient object detection method combining dynamic routing with asymmetric multi-windows is characterized in that: In step S1, , that is, both the visible light feature map and the thermal infrared feature map have 5 layers, and the feature maps of each layer of visible light are marked as ~ , the characteristic maps of each layer of thermal infrared are marked as ~ .

4. According to claim 3, a multi-modal salient object detection method combining dynamic routing with asymmetric multi-windows is characterized in that: The specific process of step S2 is: S201: The feature maps with the same number of layers in the two modalities are taken as a pair of feature maps; S202: For the three pairs of feature maps corresponding to the third layer, the second layer, and the first layer under the two modes: and , and , and Use three pairs of windows of different sizes for division; S203: Obtaining the size feature map of the visible light mode , and the size characteristic diagram of thermal infrared mode , , i =1, 2, 3.

5. According to claim 4, a multi-modal salient object detection method combining dynamic routing with asymmetric multi-windows is characterized in that: In step S202, the visible light feature map and the thermal infrared feature map of the same layer are divided into windows with the same size of windows.

6. The multi-modal salient object detection method combining dynamic routing with asymmetric multi-window according to claim 3, characterized in that: The specific process of step S301 is: S301-1: The second layer feature map of the visible light mode and thermal infrared mode and , the third layer feature map and , the fourth layer feature map and As three inputs to the routing module; S301-2: Yes and conduct Up Upsampling to get and ' ; S301-3: Yes and ' , and , and The feature maps obtained after average pooling and maximum pooling are concatenated in the feature dimension , , ; S301-4: Yes , , Flattening operation is performed to obtain ' , , ; S301-5: Yes ' , , Perform concatenation, then apply MLP and use The function obtains the routing probabilities of the second, third, and fourth layer features under the two modes.

7. The multi-modal salient object detection method combining dynamic routing with asymmetric multi-windows according to claim 6, characterized in that: In step S4, the , , and In i =1,2,3, the specific process of step S4 is: S401: Enhance the features of the first, second and third layers respectively and Flattening operation is performed to obtain and ; S402: Yes and , and Perform correlation operations to obtain aligned features of different layers and .

8. The multi-modal salient object detection method combining dynamic routing with asymmetric multi-windows according to claim 1, characterized in that: In step S5, the aligned features of different layers and In i =1,2,3, the specific process of step S5 is: S501: and Perform deformable convolution to obtain fused features , and Perform deformable convolution to obtain , and Perform deformable convolution to obtain ; S502: According to , , Get the saliency map of the preliminary prediction; S503: using the visible light zeroth layer feature map and the thermal infrared zeroth layer feature map, supplementing the preliminary predicted saliency map with detailed information; S504: Obtain the final predicted saliency map.

9. The multi-modal salient object detection method combining dynamic routing with asymmetric multi-windows according to claim 8, characterized in that: The specific process of step S502 is: S502-1: Yes After upsampling, merge; S502-2: Upsample the merged result, and merge; S502-3: Upsample the merged result at this time to obtain a preliminary predicted saliency map.

Images