Target detection method based on multi-source information fusion, thermal infrared and three-dimensional depth map

Abstract

The application discloses a target detection method based on multi-source information fusion, thermal infrared and three-dimensional depth map, and steps of the method comprise the following steps: a SalienTR model is constructed by using an encoder-decoder paradigm structure, wherein the encoder part comprises two parallel Swin Transformer backbone networks, which are respectively used for extracting appearance features of an input RGB image and spatial structure features of an input thermal infrared image or depth image; the decoder part comprises a cross-modal fusion module and a double-flow decoder, the cross-modal fusion module is used for fusing the appearance features and the spatial structure features and inputting the double-flow decoder, the double-flow decoder is used for outputting a predicted segmentation map containing a predicted salient target and an edge contour of the predicted salient target according to the input fused features and the appearance features and the spatial structure features; and the to-be-detected image and the corresponding thermal infrared image or depth image are input into the trained model, and a predicted segmentation map is output.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent manufacturing, and relates to intelligent mining and knowledge fusion of heterogeneous data, in particular to a salient object detection method of visible light, thermal infrared and three-dimensional depth map based on multi-source information fusion of Transformer. BACKGROUND

[0002] With the rapid development of big data technology and artificial intelligence technology, data mining and knowledge fusion based on massive multi-source heterogeneous data and deep learning and machine learning have become a popular research direction and are widely used in the field of intelligent manufacturing. Among them, salient object detection is a basic dense prediction technology, which is often used in downstream tasks of computer vision, such as object detection, image segmentation, object tracking, image retrieval, etc., and plays an important role in robot navigation, automatic driving, industrial product detection, automation manufacturing, etc.

[0003] In recent years, the research on salient object detection based on RGB has developed rapidly and satisfactory results have been achieved. However, under the condition of cluttered background or dim light, the texture and color details of the RGB image may be disturbed. In order to more accurately capture and locate the salient object from the complex background, researchers introduce other modal images into the salient object detection task to better supplement the missing information of the RGB image, so that the model is more robust in some challenging scenarios. For example, depth map contains more spatial and geometric information, which can highlight the outline of the object and facilitate detection; thermal infrared image reflects thermal infrared radiation above absolute zero, has more stable brightness ratio, and bad weather has little effect on thermal imaging. Since depth map and thermal infrared image provide complementary information to assist RGB image in identifying salient objects, how to effectively fuse multi-modal information has become a hot issue in the research of salient object detection.

[0004] According to the fusion strategy, the current fusion methods can be divided into early fusion, late fusion and multi-level fusion. The first two types of methods combine multi-modal information from the beginning and the end of the network respectively, and lack the interaction between different levels of features. Multi-level fusion is to interact cross-modal information between different network levels, which can fully utilize the complementarity between modalities. Among them, most of the salient object detection methods based on multi-modalities use convolutional neural networks to extract multi-modal features and use multi-level fusion to realize cross-modal information interaction, which has promoted the development of salient object detection to a great extent. However, the convolution-based method lacks the ability to capture long-distance dependencies between pixels, and performs poorly in some complex scenarios. Therefore, researchers began to introduce Transformer into this field to learn global semantic representation and further improve the accuracy of salient object detection.

[0005] According to the different cross-modal interaction modes in the Transformer, the current fusion methods can be divided into: standard self-attention, cross-attention and full joint attention. Among them, the standard self-attention implicitly establishes the relationship between different modalities, and since these relationships are relatively vague, the cross-modal complementary effect may not be ideal. Cross-attention establishes a connection between each element in one modality and all elements in another modality, but this strategy ignores the intra-modal relationship. The dense connection in the full joint attention makes up for this situation, but this redundant connection may consume a lot of resources and cause potential side effects.

[0006] Cross-modal information is the key to extract potential features and understand the overall appearance of the target, especially in some complex scene conditions such as low light and occlusion. Therefore, the core technical challenge of multi-modal fusion is to learn how to mine the complementary information between modalities. SUMMARY

[0007] In view of the above problems, the present application provides a salient object detection method based on Transformer multi-source information fusion of visible light, thermal infrared and three-dimensional depth map, referred to as SalienTR. The present application follows the typical encoder-decoder paradigm structure, in which the encoder part contains two parallel Swin Transformer backbone networks, respectively used to extract the appearance features of the RGB image and the spatial structure features of the thermal infrared or depth image; the decoder part contains a cross-modal fusion Transformer and a double-flow decoder, respectively used to fully fuse the information between multi-modalities (i.e. to fuse the appearance features and structure features) and to predict high-quality saliency map. In order to make full use of the complementary information between modalities, the present application designs a local cross-modal self-attention module in the cross-modal fusion Transformer module to capture the local feature correlation between different modalities, a global cross-modal self-attention module to establish the long-distance dependency relationship between pixels of the image and learn the global semantics, and a single-modal convolution module to effectively use the fused information and inject convolutional inductive bias. The present application includes the following steps: (1) using Swin Transformer to extract hierarchical multi-modal features of the image, i.e. appearance features and structure features; (2) using cross-modal fusion Transformer to realize multi-modal information interaction; (3) using double-flow decoder for saliency inference to predict salient objects and their edge contours.

[0008] The specific technical solutions adopted are as follows:

[0009] 1. Obtain and organize RGB-T and RGB-D salient object detection datasets for training and testing.

[0010] 1.1. The RGB-T salient object detection dataset includes three commonly used datasets, namely VT821, VT1000 and VT5000. Among them, the VT821 dataset contains 60 simple scenes, and different background information, lighting conditions and target positions are considered in the creation process; the VT1000 dataset contains 400 objects in 10 scenes, which improves the boundary of the target in the thermal infrared image; the VT5000 dataset contains 11 challenging scenes. The samples in the RGB-T salient object detection dataset of the present application have no restrictions on application scenarios, as long as the RGB images and corresponding thermal infrared images of each scene are collected.

[0011] 1.2. The collected RGB-T dataset is divided into a training set and a test set. The training set is composed of 2500 image pairs in the VT5000 dataset, and the test set is composed of the remaining 2500 image pairs in the VT821 dataset, the VT1000 dataset and the VT5000 dataset, a total of 4321 image pairs.

[0012] 1.3. The RGB-D salient object detection dataset includes nine commonly used datasets, namely NLPR, SIP, NJUD, ReDWeb, SSD, LFSD, STERE, DES and DUT. Among them, the NLPR dataset includes 1000 pairs of RGB images and depth images in 11 indoor and outdoor scenes; the SIP dataset mainly focuses on human detection and is captured by a smartphone in an outdoor scene, a total of 929 pairs of RGB images and depth images; the NJUD dataset includes 2003 pairs of RGB images and depth images from real life, three-dimensional movies and the Internet; the ReDWeb dataset and the SSD dataset include 3600 pairs and 100 pairs of RGB images and depth images, respectively; the LFSD dataset and the DUT dataset are collected by LYTRO cameras in indoor and outdoor scenes, and include 100 pairs and 1200 pairs of RGB images and depth images, respectively; the STERE dataset includes 1000 pairs of binocular images, which are obtained from the Stereoscopic Image Gallery, NVIDIA 3-Dimension Vision Live and Flickr library; the DES dataset, also known as RGB-D 135, includes 135 RGB-D image pairs in 7 indoor scenes. The samples in the RGB-D salient object detection dataset of the present application have no restrictions on application scenarios, as long as the RGB images and corresponding depth images are collected.

[0013] 1.4. The collected RGB-D dataset is divided into a training set and a test set. The training set is composed of 700 image pairs in the NLPR dataset and 1450 image pairs in NJUD, and the test set includes all samples in DES, SIP, LFSD, ReDWeb, SSD, STERE and DUT datasets and the remaining samples in the NLPR and NJUD datasets. When testing on the DUT dataset, the 800 image pairs in the DUT dataset are further put into the training set.

[0014] 1.5. The Canny edge detection method is used to process the annotated salient object segmentation map in each image in the RGB-T dataset and the RGB-D dataset, to obtain the salient object edge segmentation map corresponding to the salient object segmentation map. Finally, all the datasets include four types of images, which are RGB images, thermal infrared images or depth images, manually annotated salient object segmentation maps and generated salient object edge segmentation maps. Each pair of images and their annotation information in the dataset is used as a training sample in the training dataset, and each training sample includes the above four types of images.

[0015] 1.6. Image enhancement operations are performed on each image in each sample to expand the training samples. The images are horizontally flipped and flipped up and down; the images are subjected to affine transformation, including translation, scaling and rotation; the images are subjected to light adjustment so that the images become darker. The degree of overfitting of the model is reduced, and the detection and segmentation accuracy is effectively improved.

[0016] 2. The present application uses two parallel Swin Transformer backbone networks to extract the appearance features of the RGB image and the spatial structure features of the thermal infrared image or the depth image, respectively.

[0017] 2.1. The first Swin Transformer backbone network mainly extracts multi-level appearance features of the RGB image, and the encoded RGB features are represented as and respectively, which are down-sampled by 4, 8, 16 and 32 times, respectively.

[0018] 2.2. The second Swin Transformer backbone network mainly extracts multi-level spatial structure features of the thermal infrared image or the depth image, and the encoded thermal infrared or depth features are represented as and respectively, which are down-sampled by 4, 8, 16 and 32 times, respectively.

[0019] 2.3. When training the model, the Swin Transformer is pre-trained using the ImageNet dataset, the SalienTR model parameters are initialized using the pre-trained Swin Transformer, transfer learning is achieved, the SalienTR model converges faster, and the SalienTR model performance is improved.

[0020] 3. Based on the RGB and thermal infrared or depth features generated in step 2, the present application designs a cross-modal fusion Transformer module (Cross-modal Fusion Transformer, ComFormer) to fully fuse the information of two modalities (when training using the corresponding training samples of the RGB-T dataset, the two modalities fused are the appearance features corresponding to the RGB image and the structural features corresponding to the thermal infrared image; when training using the corresponding training samples of the RGB-D dataset, the two modalities fused are the appearance features corresponding to the RGB image and the structural features corresponding to the depth image), to prepare for the decoding process of the next stage model.

[0021] 3.1. First, the shape of the RGB feature and the thermal infrared or depth feature is adjusted to the same size through linear interpolation upsampling operation and convolution operation. Then, it is input into three weight-shared ComFormer modules for processing. The adjusted is input into a weight-shared ComFormer module, and so on.

[0022] 3.2. The first layer of ComFormer is the local cross-modal self-attention module LoC-MSA, which captures the local feature correlation between different modalities.

[0023] 3.3. The second layer of ComFormer is the global cross-modal self-attention module GLoC-MSA, which builds the global feature relationship between different modalities and learns the global semantic features.

[0024] 3.4. The third layer of ComFormer is the single-modal convolution Uni-Conv, which effectively utilizes the fused cross-modal information to further extract the features of each modality, and the extracted features inject the bias induction of convolution, improving the feature expression.

[0025] The first cross-modal fusion Transformer module first captures the local feature correlation between the appearance feature and the spatial structure feature using the local cross-modal self-attention module, and fuses it into and The global cross-modal self-attention module is used to establish the long-distance dependence between pixels in the local feature correlation And The global semantic features are learned and fused into the local feature correlation And The single-modal convolution module is used to perform feature extraction on the local feature correlation and the global semantic features And The features injected with convolution bias induction are obtained And

[0026] The second cross-modal fusion Transformer module processes the appearance features And the spatial structure features to obtain features injected with convolution bias induction And The third cross-modal fusion Transformer module processes the appearance features And the spatial structure features to obtain features injected with convolution bias induction And

[0027] 4. The fused features generated in step 3 are concatenated with the encoded features generated in step 2 in the channel dimension, and the double-flow decoder takes the concatenated features as input, goes through the decoding process, and generates a high-quality saliency segmentation result.

[0028] 4.1. The RGB features generated by the ComFormer in step 3 are concatenated in the channel dimension to generate concatenated RGB features; the thermal infrared or depth features generated by the ComFormer in step 3 are concatenated in the channel dimension to generate concatenated thermal infrared or depth features.

[0029] 4.2. The double-flow decoder takes the concatenated RGB and thermal infrared or depth features as input, uses 2 times bilinear interpolation upsampling operation and convolution operation with a receptive field of 3*3 to process them, respectively generating RGB features F *R and thermal infrared or depth features F *T .

[0030] 4.3. The RGB features F *R generated in step 4.2 are concatenated with the RGB features generated by the Swin Transformer in step 2 in the channel dimension to generate concatenated RGB features The thermal infrared or depth feature F generated in step 4.2 *T The RGB feature generated by the Swin Transformer in step 2 The thermal infrared or depth feature F generated in step 4.3

[0031] 4.4. The thermal infrared or depth feature F generated in step 4.3 The thermal infrared or depth feature F generated in step 4.3

[0032] 4.5. The RGB feature generated in step 4.3 and the thermal infrared or depth feature F The thermal infrared or depth feature F generated in step 4.3

[0033] 5. Based on the predicted segmentation map in step 4 and the real segmentation map labeled in step 1, calculate the loss function, optimize the SalienTR model iteratively, until the SalienTR model converges, and finally determine the SalienTR model parameter weight of the multi-modal salient object detection algorithm.

[0034] 5.1. Select the cross-entropy function in machine learning as the loss function of the SalienTR model.

[0035] 5.2. Calculate the loss function of the salient object segmentation map predicted in step 4 and the salient object segmentation map labeled by artificial labeling in step 1; calculate the loss function of the salient object edge segmentation map predicted in step 4 and the salient object edge segmentation map generated by Canny edge detection in step 1.

[0036] 5.3. Update the parameter weight of the SalienTR model proposed in the present application step by step through the SGD optimizer and back propagation algorithm, until the model converges, and finally determine the parameter weight of the RGB-T and RGB-D salient object detection algorithm.

[0037] 6. Test the images in step 1, use PR curve, F curve, MAE, S-Measure, F-Measure, weighted F-Measure and E-Measure evaluation indexes for algorithm evaluation, design and conduct ablation experiments, and determine the final structure of the model.

[0038] Application stage: when applying the model, input the to-be-detected image and its corresponding thermal infrared image or depth image into the trained model, sequentially pass through the Swin Transformer backbone network to extract the features of different modal images, then input the extracted features into the cross-modal fusion Transformer to realize the fusion of multi-modal features, and finally input the fused features into the double-flow decoder network to generate the final predicted segmentation map.

[0039] The advantages of the present application are as follows:

[0040] The present application designs a sparse and fully connected cross-modal fusion Transformer module, referred to as ComFormer, and based on the fusion module, a new multi-modal salient object detection framework SalienTR is constructed. The model framework can effectively extract and fuse RGB and thermal infrared or depth features to generate high-quality saliency maps and corresponding contour maps. The present application improves the fusion effect of multi-modal data and improves the precision of salient object detection, which has promoting significance for intelligent mining and knowledge fusion method research of massive heterogeneous data. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 It is a schematic diagram of the overall structure of the present application.

[0042] Figure 2 It is a schematic diagram of the ComFormer module proposed by the present application.

[0043] (a) original Transformer layer, (b) cross-modal fusion Transformer layer.

[0044] Figure 3 It is a comparison chart of the results of the present application and other RGB-T salient object detection methods.

[0045] Figure 4 It is a comparison chart of the results of the present application and other RGB-D salient object detection methods. DETAILED DESCRIPTION

[0046] The technical solutions in the examples of the present application will be clearly and completely described below in combination with the drawings in the examples of the present application. In addition, the described examples are only some of the examples of the present application, not all examples. Based on the examples in the present application, all other examples obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application. Reference is made to the accompanying drawings Figure 1 A salient object detection method based on Transformer multi-source information fusion of visible light, thermal infrared and three-dimensional depth map mainly includes the following steps:

[0047] 1. Step 2 method: The present application uses two parallel Swin Transformer backbone networks to encode features for RGB images and thermal infrared images or depth images, respectively. First, the Swin Transformer divides the input image into N non-overlapping windows, each containing S*S non-overlapping grids. Then, the Swin Transformer flattens the grids within each window and linearly maps them into local features n = 1,..., N. Next, multi-head self-attention operations are performed on the generated local features to capture local and global features of the image. The multi-head self-attention operation can be simply represented as:

[0048]

[0049] where represents the query matrix of the kth feature map, the number of matrix elements is n, which is obtained by linear projection of the input image, represents the transposed key matrix of the kth feature map, the number of matrix elements is n, which is obtained by linear projection of the input image, represents the value matrix of the kth feature map, the number of matrix elements is n, which is obtained by linear projection of the input image, PE is the relative position encoding, which is used to encode the spatial position of each grid, d k is the spatial dimension of the feature, and softmax represents the activation function softmax; for each grid in each window, the weight of self-attention can be calculated as:

[0050]

[0051] where represents the query vector of the pth grid in the kth feature map, which belongs to represents the key vector of the 1st grid in the kth feature map, which belongs to represents the key vector of the s 2 th grid in the kth feature map, pe p represents the relative position encoding of the pth grid, represents the self-attention weight of the pth grid calculated, and the output of multi-head self-attention can be calculated as:

[0052]

[0053] where represents the value vector of the p' th grid in the kth feature map, which belongs to represents the calculated self-attention output of the pth grid. After the above MSA operation, the feature conversion is further carried out by using the GELU activation function and the 2-layer perception network (Multilayer Perceptron, MLP). In addition, the Swin Transformer uses the LayerNorm layer (LN) to perform normalization operation before MSA and MLP, and adds a residual connection to combine the above two parts. Since the window is fixed and non-overlapping, MSA is only calculated independently in a local window, so this multi-head self-attention mechanism ignores the relationship between different windows. In order to improve the modeling ability, the Swin Transformer uses a moving window strategy to establish cross-window connections, so that the relationship between different grids can be explicitly established, and then the global semantic features can be captured. In addition, the Swin Transformer is composed of 4 basic Swin Transformer blocks to learn more complex semantic and multi-level feature representations. In the first stage, the number of grids is set to 4, that is, S = 4, and the corresponding features are called and With the deepening of the Swin Transformer network, the number of grids gradually decreases, and the number of feature channels gradually increases, and the feature representation of the remaining stage is and

[0054] 2. Step 3.1 method: the present application performs bilinear interpolation upsampling operation and 3*3 convolution operation on the multi-level RGB features and multi-level thermal infrared or depth features generated in step 2, so as to adjust the shape of the above features to the same size, and this process can be described as follows:

[0055]

[0056]

[0057]

[0058] wherein Conv represents 3*3 convolution operation, Up2 and Up4 represent upsampling the feature map by 2 and 4 times respectively, and Cat represents splicing the features in the modal dimension. After that, the above features will be input into the ComFormer module proposed by us to learn the relationship between different modalities.

[0059] 3. Step 3.2 Method: To obtain the feature correlation between the two modalities and integrate complementary information, a simple method is to directly calculate the attention between the corresponding two grids at different modalities and the same spatial position. However, this strategy has limited attention range, insufficient fusion, and poor complementarity. To make up for this deficiency, the invention designs a local cross-modal self-attention module LoC-MSA with a larger receptive field. The features F'2, F'3, and F'4 output by step 3.1 are input into this module, and then linear projection is used to generate query vectors, key vectors, and value vectors, followed by local cross-modal self-attention calculation, and finally output the enhanced features Its role is to capture the local feature correlation between different modalities. As shown in the accompanying Figure 2 illustrated, LoC-MSA calculates the attention between adjacent 2*S*S grids, and the attention weight and attention output can be calculated as:

[0060]

[0061]

[0062] wherein, represents the query vector of the pth grid of the mth modality in the kth feature map, which is obtained by linear projection on the output feature F'2 or F'3 or F'4, represents the key vector of the pth grid of the mth modality in the kth feature map, which is obtained by linear projection on the output feature F'2 or F'3 or F'4, represents the value vector of the p'th grid of the m'th modality in the k'th feature map, which is obtained by linear projection on the output feature F'2 or F'3 or F'4, represents the calculated local self-attention weight, represents the calculated self-attention output of the pth grid of the mth modality in the kth feature map. Due to the larger attention range, LoC-MSA provides stronger cross-modal modeling capability and obtains better local inductive representation. Formally, assuming that each local region contains 2*S*S grids, the computational complexity of LoC-MSA on a feature map with resolution H*W is: Ω(LoC-MSA) = 8HWC 2 + 8S 2 HWC, which has linear complexity when S is a constant (default setting is 3).

[0063] 4. Step 3.3 Method: Although LoC-MSA can better capture local cross-modal features, it inherently lacks global connections within and across modalities. To capture long-range dependencies, the present invention designs a global cross-modal self-attention module, GLoC-MSA, to expand the receptive field and obtain global semantic features of different modalities. As shown in FIG. 3B, GLoC-MSA takes the output of the LoC-MSA module as input, then uses linear projection to generate query vectors, key vectors, and value vectors, followed by a strategy similar to a dilated convolution to calculate global attention, and finally outputs enhanced features Figure 2

[0064]

[0065]

[0066]

[0067] 5. Step 3.4 Method: The ComFormer proposed by the present invention adopts the LoC-MSA and GLoC-MSA attention mechanisms described above, which can effectively fuse RGB and thermal infrared or depth features while capturing local and global features across modalities. To better understand the fused features, the present invention adds a single-modal convolution layer, Uni-Conv, with a receptive field of 3*3 after the LoC-MSA and GLoC-MSA modules, which outputs the global cross-modal self-attention module ​​​​​​​​​​​​The modal decoupling is performed in the channel dimension, and is split into RGB features and thermal infrared or depth features. Then, convolution operations are performed on each modal feature separately. Finally, the multi-modal features are merged in the channel dimension, and the enhanced features are output. By utilizing complementary information to learn the latent semantics in a single modality, the cross-modal information after fusion is effectively utilized to further extract the features of each modality, and a convolution bias induction is injected to improve feature expression. As shown in FIG. 2, a ComFormer layer is composed of a LoC-MSA module, a GLoC-MSA module, and two Uni-Conv modules, and a Leaky ReLU activation function is used between the two Uni-Conv modules. The method uses an LN normalization layer before each attention module and single-modal convolution, and uses a residual connection around each module. Figure 2

[0068] 6. Step 4 method: The RGB features generated by the ComFormer in step 3 are concatenated in the channel dimension to generate concatenated RGB features; the thermal infrared or depth features generated by the ComFormer in step 3 are concatenated in the channel dimension to generate concatenated thermal infrared or depth features. The process can be represented as:

[0069]

[0070]

[0071] where BConv represents a 3*3 convolution block, including a 3*3 convolution, batch normalization, and a ReLU activation function. Next, a 2x bilinear interpolation upsampling operation and a convolution operation with a receptive field of 3*3 are used to process it, respectively generating RGB features F *R and thermal infrared or depth features F *T . After that, the generated RGB features F *R are concatenated with the RGB features F generated by the Swin Transformer in step 2 in the channel dimension to generate concatenated RGB features F The generated thermal infrared or depth features F *T are concatenated with the RGB features F generated by the Swin Transformer in step 2 in the channel dimension to generate concatenated thermal infrared or depth features F The process can be represented as:

[0072]

[0073] ​​

[0074] This invention employs a multi-task learning strategy to predict salient objects and their corresponding edge contours. Since salient objects are more likely to exhibit prominent structures and boundaries in thermal infrared or depth images, this invention utilizes… This method predicts the contours of objects and generates a final salient object edge map. Specifically, it employs bilinear interpolation upsampling to recover... The resolution is reduced to H*W, and a 3*3 convolution operation is used to output the final salient target edge segmentation map M. e For salient target segmentation map prediction, and The data is concatenated along the channel dimension, then fed into a convolutional layer, followed by batch normalization and a ReLU activation function to generate the final salient object segmentation map M. s .

[0075] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A target detection method based on multi-source information fusion, thermal infrared and three-dimensional depth map, comprising the following steps: 1) constructing a SalienTR model using an encoder-decoder paradigm structure, wherein the encoder part comprises two parallel Swin Transformer backbone networks, a first Swin Transformer backbone network for extracting appearance features of an input RGB image, and a second Swin Transformer backbone network for extracting spatial structure features of an input thermal infrared image or depth image; the decoder part comprises a cross-modal fusion Transformer module and a dual-stream decoder, the cross-modal fusion Transformer module is used for fusing the appearance features and the spatial structure features and inputting the dual-stream decoder, and the dual-stream decoder is used for outputting a predicted segmentation map containing a predicted salient target and an edge contour thereof according to the input fused features and the appearance features and the spatial structure features; wherein, the cross-modal fusion Transformer module comprises a local cross-modal self-attention module, a global cross-modal self-attention module and a single-modal convolution module; the local cross-modal self-attention module is used for capturing local feature correlation between the appearance features and the spatial structure features; the global cross-modal self-attention module is used for constructing a global feature relationship between the appearance features and the spatial structure features and learning global semantic features; and the single-modal convolution module is used for extracting features from the appearance features and the spatial structure features fused with the local feature correlation and the global semantic features, respectively, to obtain appearance features and spatial structure features injected with convolution bias induction; The double-flow decoder sequentially performs upsampling operation and convolution operation on the input RGB feature to generate an RGB feature F *R ; the double-flow decoder sequentially performs upsampling operation and convolution operation on the input thermal infrared feature or depth feature to generate a corresponding feature F *T ; the feature F *T is a thermal infrared feature or a depth feature; the RGB feature F *R is concatenated with the feature F in the channel dimension to generate an RGB feature F ; the feature F *T is concatenated with the feature F in the channel dimension to generate a feature F ; and then the feature F is sequentially subjected to upsampling operation and convolution operation to generate a predicted saliency target edge segmentation map; the RGB feature F is concatenated with the feature F in the channel dimension, and the resultant feature is sequentially subjected to upsampling operation and convolution operation to generate a predicted saliency target segmentation map; 2) obtaining an RGB-T salient target detection dataset and an RGB-D salient target detection dataset and labeling each image therein to generate a training dataset; wherein each pair of images in the RGB-T salient target detection dataset comprises an RGB image and a corresponding thermal infrared image of an application scenario, and each pair of images in the RGB-D salient target detection dataset comprises an RGB image and a corresponding depth image of an application scenario; 3) training and optimizing the SalienTR model using the training dataset; 4) inputting a to-be-detected image and its corresponding thermal infrared image or depth image into the trained SalienTR model to output a predicted segmentation map.

2. The method of claim 1, wherein, The first Swin Transformer backbone network mainly extracts appearance features of the RGB image and encodes the appearance features at different scales, and the multi-level features obtained by the encoding are represented as and The second Swin Transformer backbone network extracts spatial structure features of the thermal infrared image or the depth image and encodes the spatial structure features at different scales, and the multi-level features obtained by the encoding are represented as and The decoder part comprises three weight-shared cross-modal fusion Transformer modules. characteristics and characteristics to the same size, the adjusted input into a first cross-modal fusion Transformer module; the adjusted input into a second cross-modal fusion Transformer module, the adjusted input into a third cross-modal fusion Transformer module; The fused features output by the three cross-modal fusion Transformer modules are then fused and input into the dual-stream decoder; The bi-flow decoder generates the fusion feature according to the inputted fusion feature and An output prediction segmentation map is generated.

3. The method of claim 2, wherein, The method for obtaining the fused features is as follows: 11) The first cross-modal fusion Transformer module first uses a local cross-modal self-attention module to capture appearance features. Spatial structural features Local feature correlations between them, and integrate them into and Then, a global cross-modal self-attention module is used to establish a model that integrates the correlations of local features. and The long-range dependencies between pixels are learned, and global semantic features are fused into the model. and Then, the single-modal convolution module is used to process the data that fuses local feature correlations and global semantic features. and Feature extraction is performed to obtain features infused with convolutional bias induction. and 12) the second cross-modal fusion Transformer module processes the appearance features and the spatial structure features according to the method of step 11) to obtain features injected with convolution bias induction and the third cross-modal fusion Transformer module processes the appearance features and the spatial structure features according to the method of step 11) to obtain features injected with convolution bias induction and 13) the feature The RGB feature is obtained by splicing and fusing in the channel dimension, and the feature The corresponding thermal infrared feature or depth feature is obtained by splicing and fusing in the channel dimension.

4. The method according to claim 1 or 2 or 3, characterized in that, The labeling information comprises a salient target segmentation map and a salient target edge segmentation map, and each pair of image and labeling information is taken as a training sample in the training dataset.

5. The method of claim 1, wherein, The loss function used for training the SalienTR model is a cross-entropy function.

6. A server, characterized by A computer program product comprising a memory storing a computer program configured to be executed by a processor, the computer program comprising instructions for performing the steps of any one of claims 1 to 5. A computer program product comprising a memory storing a computer program configured to be executed by a processor, the computer program comprising instructions for performing the steps of any one of claims 1 to 5.

7. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, which is executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

Images