Salient object detection system and method based on attention mechanism and cross-modal fusion

Abstract

The present invention discloses a salient target detection system and method based on attention mechanism and cross-modal fusion, which belongs to the field of computer vision, and includes: first, a multi-scale residual attention module is proposed to extract rich and important global appearance and detail information from the RGB branch, and to capture important spatial structure information from the depth branch. Secondly, from the perspective of cross-modality of the same scale and different scales, a dense fusion module is proposed to fuse the extracted high-level RGB features and depth features to generate an initial salient map. Finally, the boundary refinement module proposed uses the low-level features of the two branches to refine the boundaries of the targets in the initial salient map, so that the final salient map contains sharp boundaries and complete salient targets.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a salient object detection system and method based on attention mechanism and multi-scale cross-modal fusion. Background Art

[0002] Salient object detection aims to simulate the human visual perception system to autonomously detect the most important and attractive salient objects in a given image. It has been applied in various computer vision tasks, such as image segmentation, visual tracking, object recognition, image retrieval, etc. In recent years, with the rapid development of deep learning, RGB-D salient object detection has made great progress. However, when dealing with challenging scene images, such as cluttered background, similar foreground and background, low contrast, etc., some existing methods cannot fully detect salient objects in the image.

[0003] In the field of RGB-D salient object detection, RGB images contain rich appearance cues and position information, while depth images provide useful spatial structure and 3D layout information, which are important complementary information to the appearance cues in RGB images. In order to detect and obtain a complete salient map, we need to consider two important issues: 1) how to extract rich and important feature information from RGB images and depth images respectively, and 2) how to perform effective multi-scale cross-modal fusion between RGB features and depth features.

[0004] At present, salient object detection methods mainly face the following difficulties:

[0005] 1. Regarding feature extraction, most current detection methods do not take into account the large amount of feature information extracted by the backbone network at each layer and the redundancy problem, and do not consider the differences in space and channels of the extracted feature information, but use it directly.

[0006] 2. Regarding feature fusion, most of the existing fusion strategies are divided into three types: early fusion, mid-term fusion, and late fusion. The early fusion method does not consider the difference between the two images, and direct splicing will destroy the inherent internal characteristics of the two images, which will eventually lead to unsatisfactory prediction results; the late fusion method does not focus on the hierarchical information complementarity between the appearance information contained in the RGB image and the spatial structure information contained in the depth image, and directly fusing the final result cannot fully explore the information correlation between the modalities; while the mid-term fusion can effectively solve the difference problem between the two modalities and fully explore the complementarity between the two modalities.

[0007] Currently, a lot of research has been done on salient object detection and many methods have been proposed. Reference Y.Piao, W.Ji, J.Li, M.Zhang, and H.Lu, "Depth-Induced Multi-Scale Recurrent Attention Networkfor Saliency Detection," in 2019IEEE / CVF International Conference on Computer Vision (ICCV) proposed a deep refinement module to fuse multi-level complementary RGB and depth features, and then sent the fused features to the depth-induced multi-scale weighted module and the recurrent attention module for saliency detection; Reference Z.Liu, S.Shi, Q.Duan, W.Zhang, and P.Zhao.Salient object detection for RGB-D image by singlestream recurrent convolution neural network.Neurocomputing, 363:46–57, 2019 RGB and depth images are concatenated as four-channel inputs to the backbone network to generate multiple layers of features, and then the deep recurrent convolutional neural network is used to pre-process each layer of features from depth to depth, and finally the saliency map of each layer is fused to produce the final result; Reference N.Wang and X.Gong.Adaptive fusion for RGB-Dsalient object detection.IEEE Access,7:55277–55284,2019 The RGB and depth images are respectively sent into the RGB salient prediction stream and the depth salient prediction stream to obtain two salient maps, and then a saliency fusion module is used to produce the final result; the literature Hao Chen,Youfu Li,and DanSu.Multi-modal fusion network with multi-scale multi-path and cross-modalinteractions for rgb-d salient object detection.Pattern Recognition,86:376–385,2019 proposed a multi-scale multi-path fusion cross-modal interaction network for salient object detection.

[0008] The above methods either fail to consider the redundancy of the extracted feature information and lack of focus, or fail to consider the internal differences between RGB images and depth images, or fail to fully explore the complementarity between the two modalities. Therefore, the detected salient maps do not have a relatively complete prominent target. Summary of the invention

[0009] The present invention provides a salient target detection system and method based on attention mechanism and multi-scale cross-modal fusion, aiming to solve the technical problems that existing methods have redundancy and lack of emphasis in extracting feature information and insufficient interaction between modalities.

[0010] In view of the above problems in the prior art, according to one aspect disclosed in the present invention, the present invention adopts the following technical solution:

[0011] A salient object detection method based on attention mechanism and multi-scale cross-modal fusion, comprising:

[0012] S1. The color image and depth image are respectively sent to the backbone network to extract hierarchical features of different scales of the two modalities;

[0013] S2. The hierarchical features of different scales extracted by the backbone network are fed into the multi-scale residual attention module, and feature information including high three-layer features and low-layer features are extracted from the color branch and the depth branch respectively;

[0014] S3. sending the high three-layer features to a dense fusion module to explore complementary information between modalities and utilizing them to generate an initial saliency map;

[0015] S4. A boundary refinement module uses the low-level features extracted from the two branches to refine the salient objects in the initial saliency map and produce the final saliency map.

[0016] In order to better realize the present invention, a further technical solution is:

[0017] Furthermore, the backbone network adopts ResNet.

[0018] Furthermore, the ResNet adopts ResNet50.

[0019] Furthermore, in the step S1, the method of extracting hierarchical features of different scales from the color image and the depth image respectively includes:

[0020] The resolution of color image and depth image input to the backbone network is 256x256;

[0021] After the five convolutional blocks {Conv1, Conv2, Conv3, Conv4, Conv5} of the backbone network, the side output features of the two branches {R1 / D1, R2 / D2, R3 / D3, R4 / D4, R5 / D5} are obtained, and the corresponding resolutions are {1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32} of the input image resolution respectively;

[0022] A 1x1 convolutional layer is applied to the side output features obtained from the two branches to reduce the channel dimension to {64, 64, 64, 64, 64}.

[0023] Furthermore, in the step S2, the method of sending the hierarchical features of different scales extracted by the backbone network to the multi-scale residual attention module to extract feature information includes:

[0024] R i =MSRA(r i )i=1,2,3,4,5

[0025] D i =MSRA(d i )i=1,2,3,4,5

[0026] Among them, MSRA stands for Multi-Scale Residual Attention Module, R i and D i They represent the RGB features and depth features output by the multi-scale residual attention module respectively.

[0027] Furthermore, in the step S3, the method of sending the high three-layer features to the dense fusion module to generate the initial saliency map includes:

[0028] First, perform cross-modal fusion of the same scale on the top three layers of RGB features {R3, R4, R5} and depth features {D3, D4, D5}, and send the features of the same scale into the dense block to obtain three mixed feature maps, which are then integrated to obtain a fused feature map F fuse1 , the specific operations are as follows:

[0029] F ij =Dense(R i , D j ), i = j∈{3, 4, 5}

[0030] F fuse1 =DFM1(F 33 , F 44 , F 55 )

[0031] In the above formula, Dense represents a dense block, and DFM1 represents a dense fusion module of the same scale fusion;

[0032] Secondly, cross-modal fusion of different scales is performed on the three-layer RGB features {R3, R4, R5} and the depth features {D3, D4, D5}. The features of different scales are combined in pairs through dense blocks to obtain 6 mixed feature maps, which are then passed through two dense fusion modules to generate two feature maps F fuse2 and F fuse3 , the specific operations are as follows:

[0033] F ij =Dense(R i , D j ), i=j∈({3, 4, 5} and i≠j

[0034] F fuse2 =DFM2(F 34 , F 35 , F 45 )

[0035] F fuse3 =DMF2(F 43 , F 53 , F 54 )

[0036] In the above formula, DMF2 represents the dense fusion module for fusion of different scales;

[0037] Finally, the three fusion features are added together to produce an initial saliency map, namely:

[0038]

[0039] In the above formula, Represents the element addition operation.

[0040] Furthermore, in step S4, the method of generating the final saliency map includes:

[0041] First, the following operations are performed on the low-level RGB features {R1, R2} and deep features {D1, D2} extracted by the multi-scale residual attention module to generate two feature maps F1 and F2:

[0042]

[0043] In the above formula, Represents the element addition operation;

[0044] Then, the two feature maps generated by the above formula are used to refine the initial saliency map to generate two refined feature maps and The operation is as follows:

[0045]

[0046] In the above formula, and They represent element addition and element multiplication operations respectively, and UP represents upsampling operation;

[0047] Finally, the two refined feature maps and Perform splicing operation to generate the final saliency map S fin, which operates as follows:

[0048]

[0049] In the above formula, UP, Cat, and Conv represent upsampling operation, concatenation operation, and a 3×3 convolutional layer, respectively.

[0050] The present invention may also be:

[0051] A salient object detection system based on attention mechanism and multi-scale cross-modal fusion, comprising:

[0052] The feature extraction module uses the backbone network to extract hierarchical features of different scales for color images and depth images;

[0053] A multi-scale residual attention module is used to extract feature information including high-level features and low-level features from the color branch and the depth branch respectively;

[0054] A dense fusion module, for exploring complementary information between modalities based on the high three-layer features, and utilizing the information to generate an initial saliency map;

[0055] The boundary refinement module is used to refine the salient objects in the initial saliency map from the low-level features extracted from the two branches to produce the final saliency map.

[0056] Furthermore, the dense fusion module includes:

[0057] The same-scale cross-modal fusion is performed on the three-layer RGB features {R3, R4, R5} and the depth features ({D3, D4, D5}), and the features of the same scale are sent to the dense block to obtain three mixed feature maps, which are then integrated to obtain a fused feature map F fuse1 , the specific operations are as follows:

[0058] F ij =Dense(R i , D j ), i = j∈{3, 4, 5}

[0059] F fuse1 =DFM1(F 33 , F 44 , F 55 )

[0060] In the above formula, Dense represents a dense block, and DFM1 represents a dense fusion module of the same scale fusion;

[0061] The high three-layer RGB features ({R3, R4, R5}) and depth features ({D3, D4, D5}) are fused at different scales. The features at different scales are combined in pairs through dense blocks to obtain 6 mixed feature maps, which are then passed through two dense fusion modules to generate two feature maps F fuse2 and F fuse3 , the specific operations are as follows:

[0062] F ij =D ense (R i , D j ), i=j∈{3, 4, 5} and i≠j

[0063] F fuse2 =DFM2(F 34 , F 35 , F 45 )

[0064] F fuse3 =DMF2(F 43 , F 53 , F 54 )

[0065] In the above formula, DMF2 represents the dense fusion module for fusion of different scales;

[0066] Add the three fusion features obtained to produce an initial saliency map, namely:

[0067]

[0068] In the above formula, Represents the element addition operation.

[0069] Furthermore, the boundary refinement module includes:

[0070] The following operations are performed on the low-level RGB features {R1, R2} and deep features {D1, D2} extracted by the multi-scale residual attention module to generate two feature maps F1 and F2, which are:

[0071]

[0072] In the above formula, Represents the element addition operation;

[0073] Use the two feature maps generated by the above formula to refine the initial saliency map and generate two refined feature maps and The operation is as follows:

[0074]

[0075] In the above formula, and They represent element addition and element multiplication operations respectively, and UP represents upsampling operation;

[0076] For the two refined feature maps and Perform splicing operation to generate the final saliency map S fin , which operates as follows:

[0077]

[0078] In the above formula, UP, Cat, and Conv represent upsampling operation, concatenation operation, and a 3×3 convolutional layer, respectively.

[0079] Compared with the prior art, one of the beneficial effects of the present invention is:

[0080] The present invention is a salient target detection method and system based on attention mechanism and multi-scale cross-modal fusion. By adding the proposed multi-scale residual attention module to extract rich and important feature information from the backbone network, it solves the problem of redundancy and lack of focus in extracting feature information in previous methods. In addition, the proposed intermediate fusion strategy consisting of dense fusion module and boundary refinement module fully mines the complementary information between modalities and avoids the problems existing in early fusion and late fusion strategies. When detecting scene images with challenges such as similar foreground and background, multiple targets, and cluttered background, the present invention can detect more complete salient targets and sharp boundary details compared with previous methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the embodiments of the present application document or the technical solutions in the prior art, the drawings required for use in the description of the embodiments or the prior art are briefly introduced below. Obviously, the drawings described below are only references to some embodiments in the present application document. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0082] Figure 1 FIG. 4 is a schematic diagram of the overall network structure according to an embodiment of the present invention.

[0083] Figure 2 Schematic diagram of a multi-scale residual attention module according to an embodiment of the present invention.

[0084] Figure 3a Schematic diagram of dense fusion modules of the same scale according to an embodiment of the present invention.

[0085] Figure 3b Schematic diagram of dense fusion modules of different scales according to an embodiment of the present invention.

[0086] Figure 4 FIG. 4 is a schematic diagram of a boundary refinement module according to an embodiment of the present invention.

[0087] Figure 5 FIG. 4 is a schematic diagram showing comparison results between a salient object detection method according to an embodiment of the present invention and an existing method. DETAILED DESCRIPTION

[0088] The present invention is further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

[0089] like Figures 1 to 4 As shown, a salient object detection method based on attention mechanism and multi-scale cross-modal fusion includes the following steps:

[0090] S1. The color image and the depth image are respectively sent to the backbone network to extract hierarchical features of different scales of the two modalities; the backbone network is preferably a ResNet network, and more preferably a ResNet50;

[0091] S2. sending the hierarchical features of different scales extracted from the two images by the backbone network into the multi-scale residual attention module, respectively extracting rich and important features from the color branch and the depth branch, the rich and important features containing feature information of high three-layer features and low-layer features;

[0092] S3. Send the above-extracted rich and important high-level three-layer features to the dense fusion module to explore the complementary information between modalities and use them to generate an initial saliency map;

[0093] S4. A boundary refinement module uses the low-level features extracted from the two branches to refine the salient objects in the initial saliency map and produce the final saliency map.

[0094] In step S1 of the above embodiment, the backbone network is used to extract hierarchical features of different scales from the RGB image and the depth image respectively, specifically:

[0095] The resolution of the color image and depth image input into the backbone network is 256x256, and ResNet50 is generally used as the backbone network to extract multi-layer RGB features and depth features;

[0096] After the five convolutional blocks {Conv1, Conv2, ConV3, Conv4, Conv5} of the backbone network, the side output features of the two branches {R1 / D1, R2 / D2, R3 / D3, R4 / D4, R5 / D5} are obtained, and the corresponding resolutions are {1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32} of the input image resolution respectively;

[0097] To facilitate the implementation of subsequent module functions, 1×1 convolutional layers are applied to the RGB branch and depth branch respectively to reduce their channel dimensions to {64, 64, 64, 64, 64}.

[0098] In step S2 of the above embodiment, the method of sending the hierarchical features of different scales extracted by the backbone network to the multi-scale residual attention module to extract feature information includes:

[0099] R i =MSRA(r i )i=1,2,3,4,5

[0100] D i =MSRA(d i )i=1,2,3,4,5

[0101] Among them, MSRA stands for Multi-Scale Residual Attention Module, R i and D i They represent the RGB features and depth features output by the multi-scale residual attention module respectively.

[0102] For example Figure 2 As shown, the RGB feature r of the input multi-scale residual attention module i and the deep feature d i , first pass through 5 branches, from top to bottom, and output a feature map r through the first branch i 1 / d i 1 , as follows:

[0103]

[0104] Among them, conv1 and conV3 represent 1x1 and 3x3 convolutional layers respectively.

[0105] After the second, third, and fourth branches, each branch obtains a feature map, as follows:

[0106] α i =CA(r i / d i )β i =SA(r i / d i )

[0107]

[0108]

[0109]

[0110] Among them, CA and SA represent channel attention and spatial attention respectively, a i and β i They represent the weighted feature maps of the input features after channel attention and spatial attention, respectively. conv1, conv3, conv5 and conv7 represent the convolutional layers of 1x1, 3x3, 5x5 and 7x7, respectively. d=3, 5, 7 represent the convolution expansion rates of 3, 5, 7, respectively.

[0111] The fifth branch consists of only one 1x1 convolutional layer to reduce the number of channels, namely:

[0112]

[0113] Finally, the first four branches are concatenated and added to the last branch to form a residual structure to obtain the output features, namely:

[0114]

[0115] In the above formula, Cat represents the splicing operation. Represents the element addition operation.

[0116] In the above step S3, the high three layer features are sent to the dense fusion module to generate the initial saliency map, specifically:

[0117] First, dense fusion modules of the same scale such as Figure 3a As shown in Figure 1, the extracted high-level RGB features {R3, R4, R5} and deep features {D3, D4, D5} are fused through three classic dense blocks to generate three mixed feature maps of the same scale, which are then integrated into a fused feature map F fuse1 , the specific operations are as follows:

[0118] F ij =Dense(R i , D j ), i = j∈{3, 4, 5}

[0119] F fuse1 =DFM1(F 33 , F 44 , F 55 )

[0120] In the above formula, Dense represents a dense block, and DFM1 represents a dense fusion module of the same scale fusion; through the above formula, three feature maps {F 33 , F 44 , F 55 ), then do the following:

[0121]

[0122]

[0123]

[0124]

[0125] In the above formula, UP2 and UP4 represent 2x and 4x upsampling operations, and Conv represents a 3×3 convolution operation. represents the element addition operation, and Cat represents the concatenation operation.

[0126] Secondly, dense fusion modules of different scales such as Figure 3b As shown in the figure, for the extracted high three-layer RGB features {R3, R4, R5) and depth features {D3, D4, D5}, six classic dense blocks are used to perform cross-modal fusion of different scales to generate six mixed feature maps, which are then integrated into two fused feature maps F fuse2 and F fuse3 , the specific operations are as follows:

[0127] F ij =Dense(R i , D j ), i=j∈{3, 4, 5} and i≠j

[0128] In the above formula, Dense represents a dense block. Through the above formula, we will get 6 feature maps (F 34 , F 35 , F 45 , F 43 , F 53 , F 54 ), then we integrate the first three and the last three feature maps respectively to obtain two fused feature maps. The specific operations are as follows:

[0129]

[0130]

[0131]

[0132]

[0133] In the above formula, UP2 represents a 2x upsampling operation, and Conv represents a 3×3 convolution operation. Represents element addition operation, and Cat represents concatenation operation. Similarly, the last three feature maps can also be used to obtain a fused feature map F through the above operation. fuse3 .

[0134] Finally, the initial saliency map Sini As shown below:

[0135]

[0136] In the above formula Represents an element-wise addition operation.

[0137] In the above step S4, the initial saliency map is refined by extracting rich and important low-level features to obtain the final saliency map, which is specifically:

[0138] Boundary refinement module such as Figure 4 As shown in Figure 1, first, the following operations are performed on the low-level RGB features {R1, R2} and deep features {D1, D2} extracted by the multi-scale residual attention module to generate two feature maps F1 and F2:

[0139]

[0140] In the above formula, Represents the element addition operation.

[0141] Then, the two feature maps generated by the above formula are used to refine the initial saliency map to generate two refined feature maps and The operation is as follows:

[0142]

[0143] In the above formula, and They represent element-wise addition and element-wise multiplication operations respectively, and UP represents upsampling operation.

[0144] Finally, the two refined feature maps and Perform splicing operation to generate the final saliency map S fin , which operates as follows:

[0145]

[0146] In the above formula, UP, Cat and Conv represent upsampling operation, concatenation operation and a 3×3 convolutional layer respectively.

[0147] For the implementation of the above-mentioned salient object detection method, the present invention also discloses a salient object detection system based on attention mechanism and multi-scale cross-modal fusion, which includes:

[0148] The feature extraction module uses the backbone network to extract hierarchical features of different scales for color images and depth images;

[0149] A multi-scale residual attention module is used to extract feature information including high-level features and low-level features from the color branch and the depth branch respectively;

[0150] A dense fusion module, for exploring complementary information between modalities based on the high three-layer features, and utilizing the information to generate an initial saliency map;

[0151] The boundary refinement module is used to refine the salient objects in the initial saliency map from the low-level features extracted from the two branches to produce the final saliency map.

[0152] And the functional units implementing the methods in the above embodiments are correspondingly arranged in the above modules.

[0153] In this invention, the effectiveness and advantages of the proposed method are demonstrated by comparing it with existing salient object detection models. The compared models include:

[0154] (1)H.Chen and Y.Li.Three-stream attention-aware network for rgb-dsalient object detection. IEEE Trans.Image Process., 28: 2825-2835, 2019.

[0155] (2)Jia-Xing Zhao, Yang Cao, Deng-Ping Fan, Ming-Ming Cheng, Xuan-Yi Li, and Le Zhang.Contrast prior and fluid pyramid integration for rgbd salientobject detection.In IEEE Conf.Comput.Vis.Pattern Recog., 2019.

[0156] (3) Yongri Piao, Wei Ji, Jingjing Li, Miao Zhang, and Huchuan Lu. Depth-induced multi-scale recurrent attention network for saliency detection. InInt.Conf.Comput.Vis., pages7254-7263, 2019.

[0157] (4)Li G, Liu Z and Ling H.ICNet: Information Conversion Network forRGB-D Based Salient Object Detection.IEEE Transactions on Image Processing, pages 4873-4884, 2020.

[0158] (5)Yongri Piao, Zhengkun Rong, Miao Zhang, W.Ren, and Huchuan Lu.A2dele: Adaptive and attentive depth distiller for efficient rgb-d salient object detection. IEEE Conf.Comput.Vis.Pattern Recog., pages 9057–9066, 2020.

[0159] (6)Nian Liu, N. Zhang, and J. Han. Learning selective self-mutualattention for rgb-d saliency detection. IEEE Conf. Comput. Vis. Pattern Recog., pages 13753–13762, 2020.

[0160] 1485 samples from NJUD, 700 samples from NLPR and 800 samples from DUT are selected for all model training, and the remaining samples from NJUD, NLPR and DUT as well as the entire LFSD and SIP datasets are used for testing. For all indicators, the salient object detection method in the present invention shows good performance. Under four evaluation indicators and five challenging datasets, a large number of experiments confirm that the model architecture proposed by us is superior to the current advanced salient object detection method. The specific parameter comparison results are shown in Figure 2. Figure 5 shown.

[0161] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0162] References to "one embodiment", "another embodiment", "embodiment", etc. in this specification refer to specific features, structures or characteristics described in conjunction with the embodiment as included in at least one embodiment generally described in this application. The appearance of the same expression in multiple places in the specification does not necessarily refer to the same embodiment. Further, when a specific feature, structure or characteristic is described in conjunction with any embodiment, it is claimed that the realization of such feature, structure or characteristic in conjunction with other embodiments also falls within the scope of the present invention.

[0163] Although the present invention is described herein with reference to a plurality of illustrative embodiments of the present invention, it should be understood that those skilled in the art may devise many other modifications and implementations that fall within the scope and spirit of the principles disclosed herein. More specifically, within the scope of the present disclosure and claims, a variety of variations and improvements may be made to the components and / or layout of the subject combination layout. In addition to variations and improvements made to the components and / or layout, other uses will also be apparent to those skilled in the art.

Claims

1. A salient object detection method based on attention mechanism and multi-scale cross-modal fusion, characterized in that include: S1. The color image and depth image are respectively sent to the backbone network to extract hierarchical features of different scales of the two modalities; S2. The hierarchical features of different scales extracted by the backbone network are fed into the multi-scale residual attention module, and feature information including high three-layer features and low-layer features are extracted from the color branch and the depth branch respectively; S3. sending the high three-layer features to a dense fusion module to explore complementary information between modalities and utilizing them to generate an initial saliency map; In the step S3, the method of sending the high three-layer features to the dense fusion module to generate the initial saliency map includes: First, perform cross-modal fusion of the same scale on the top three layers of RGB features {R3, R4, R5} and depth features {D3, D4, D5}, and send the features of the same scale into the dense block to obtain three mixed feature maps, which are then integrated to obtain a fused feature map F fuse1 , the specific operations are as follows: F ij =Dense(R i ,D j ),i=j∈{3,4,5} F fuse1 =DFM1(F 33 ,F 44 ,F 55 ) In the above formula, Dense represents a dense block, and DFM1 represents a dense fusion module of the same scale fusion; Secondly, cross-modal fusion of different scales is performed on the three-layer RGB features {R3, R4, R5} and the depth features {D3, D4, D5}. The features of different scales are combined in pairs through dense blocks to obtain 6 mixed feature maps, which are then passed through two dense fusion modules to generate two feature maps F fuse2 and F fuse3 , the specific operations are as follows: F ij =Dense(R i , D j ), i=j∈{3, 4, 5} and i≠j F fuse2 =DFM2(F 34 ,F 35 ,F 45 ) F fuse3 =DMF2(F 43 ,F 53 ,F 54 ) In the above formula, DMF2 represents the dense fusion module for fusion of different scales; Finally, the three fusion features are added together to produce an initial saliency map, namely: In the above formula, Represents the element addition operation; S4. Refining the salient objects in the initial salient map using the low-level features extracted from the two branches through a boundary refinement module to generate a final salient map; in the step S4, the method of generating the final salient map includes: First, the following operations are performed on the low-level RGB features {R1, R2} and deep features {D1, D2} extracted by the multi-scale residual attention module to generate two feature maps F1 and F2: In the above formula, Represents the element addition operation; Then, the two feature maps generated by the above formula are used to refine the initial saliency map to generate two refined feature maps and The operation is as follows: In the above formula, and They represent element addition and element multiplication operations respectively, and UP represents upsampling operation; Finally, the two refined feature maps and Perform splicing operation to generate the final saliency map S fin , which operates as follows: In the above formula, UP, Cat, and Conv represent upsampling operation, concatenation operation, and a 3×3 convolutional layer, respectively.

2. The salient object detection method based on attention mechanism and multi-scale cross-modal fusion according to claim 1 is characterized in that The backbone network adopts ResNet.

3. The salient object detection method based on attention mechanism and multi-scale cross-modal fusion according to claim 2 is characterized in that The ResNet adopts ResNet50.

4. The salient object detection method based on attention mechanism and multi-scale cross-modal fusion according to claim 1 is characterized in that In the step S1, the method of extracting hierarchical features of different scales from the color image and the depth image respectively includes: The resolution of the color image and depth image input to the backbone network is 256х256; After the five convolutional blocks {Conv1, Conv2, Conv3, Conv4, Conv5} of the backbone network, the side output features of the two branches {R1 / D1, R2 / D2, R3 / D3, R4 / D4, R5 / D5} are obtained, and the corresponding resolutions are {1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32} of the input image resolution respectively; A 1х1 convolutional layer is applied to the side output features obtained from the two branches to reduce the channel dimension to {64, 64, 64, 64, 64}.

5. The salient object detection method based on attention mechanism and multi-scale cross-modal fusion according to claim 1 is characterized in that In the step S2, the hierarchical features of different scales extracted by the backbone network are sent to the multi-scale residual attention module to extract feature information, including: R i =MSRA(r i ) i=1,2,3,4,5 IS i =MSRA(d i ) i=1,2,3,4,5 Among them, MSRA stands for Multi-Scale Residual Attention Module, R i and D i They represent the RGB features and depth features output by the multi-scale residual attention module respectively.

6. A salient object detection system based on attention mechanism and multi-scale cross-modal fusion, characterized by include: The feature extraction module uses the backbone network to extract hierarchical features of different scales for color images and depth images; A multi-scale residual attention module is used to extract feature information including high-level features and low-level features from the color branch and the depth branch respectively; A dense fusion module, for exploring complementary information between modalities based on the high three-layer features, and utilizing the information to generate an initial saliency map; The dense fusion module includes: Perform cross-modal fusion of the same scale on the three-layer RGB features {R3, R4, R5} and the depth features {D3, D4, D5}, send the features of the same scale into the dense block, and obtain three mixed feature maps. After integration, a fused feature map F is obtained. fuse1 , the specific operations are as follows: F ij =Dense(R i ,D j ),i=j∈{3,4,5} F fuse1 =DFM1(F 33 ,F 44 ,F 55 ) In the above formula, Dense represents a dense block, and DFM1 represents a dense fusion module of the same scale fusion; The high three-layer RGB features {R3, R4, R5} and the depth features {D3, D4, D5} are fused at different scales. The features at different scales are combined in pairs through dense blocks to obtain 6 mixed feature maps, which are then passed through two dense fusion modules to generate two feature maps F fuse2 and F fuse3 , the specific operations are as follows: F ij =Dense(R i , D j ), i=j∈{3, 4, 5} and i≠j F fuse2 =DFM2(F 34 ,F 35 ,F 45 ) F fuse3 =DMF2(F 43 ,F 53 ,F 54 ) In the above formula, DMF2 represents the dense fusion module for fusion of different scales; Add the three fusion features obtained to produce an initial saliency map, namely: In the above formula, Represents the element addition operation; A boundary refinement module is used to refine the salient objects in the initial salient map from the low-level features extracted from the two branches to generate a final salient map; the boundary refinement module includes: The following operations are performed on the low-level RGB features {R1, R2} and deep features {D1, D2} extracted by the multi-scale residual attention module to generate two feature maps F1 and F2, which are: In the above formula, Represents the element addition operation; Use the two feature maps generated by the above formula to refine the initial saliency map and generate two refined feature maps and The operation is as follows: In the above formula, and They represent element addition and element multiplication operations respectively, and UP represents upsampling operation; For the two refined feature maps and Perform splicing operation to generate the final saliency map S fin , which operates as follows: In the above formula, UP, Cat, and Conv represent upsampling operation, concatenation operation, and a 3×3 convolutional layer, respectively.

Images