An RGB-D salient object detection method based on boundary deformable convolution guidance

Abstract

The application discloses an RGB-D salient object detection method based on boundary deformable convolution guidance, comprising the following steps: step one, respectively extracting features of an RGB mode and a depth map mode; step two, fusing the features of the two modes through a cross-modal attention fusion feature module to mine common and complementary features of salient objects; step three, inputting the feature map into an encoder deep layer embedded with an adjacent multi-scale feature enhancement module to obtain global context feature information; step four, generating a boundary clue map of the salient objects by constructing a boundary feature extraction module; and step five, generating a saliency map by using the generated boundary clue map and deformable convolution guidance. The application mines and strengthens the commonness of salient objects by cross-fusion of the depth map and the RGB image, effectively captures salient objects with different sizes and uncertain quantities by using adjacent level feature interaction, and solves the boundary blur problem of the saliency map by using the edge clue map to guide the model decoding.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an RGB-D salient object detection method based on boundary deformable convolution guidance. BACKGROUND

[0002] Salient object detection (SOD) aims to segment the most eye-catching objects or regions in a natural image. It simulates the human visual attention mechanism to process only specific regions in the image, which can improve the computing efficiency and save the computing and storage costs.

[0003] Many methods have achieved good detection effects by taking RGB data as the research task, but it is difficult to accurately detect complete targets in low-light, background and salient object color or similar scenes. Depth map has good geometric structure, internal consistency and illumination invariance, which can provide auxiliary information for RGB image and improve the performance of model detection.

[0004] However, the prior art still has some problems in using depth map to provide auxiliary information for RGB image and improve the performance of model detection.

[0005] 1. The salient features of the depth map are relatively simple, and the low-quality depth map has more noise. Although the RGB image has high clarity, it may cause errors in the model detection result in the case where the salient object and the background have little difference in color and shape. There is still a lack of technology that effectively combines the two and effectively improves the detection effect by using the advantages of the two.

[0006] 2. The size and number of salient objects are usually different, and when the image to be detected contains multiple targets or small targets, the phenomenon of missed detection may occur.

[0007] 3. The boundary contour of the salient object in the natural scene is relatively complex, and the existing method is usually difficult to effectively identify the target structure and contour details, resulting in blurred and incomplete detection of the salient map boundary. When the boundary contour features of the salient object are not obvious enough, it cannot be distinguished from the surrounding background region, resulting in the background being misjudged as a salient region or the salient object being misjudged as the background.

[0008] The deep feature map contains rich salient target position information, a large convolution kernel can obtain a large receptive field, which is beneficial to capture the features of large targets, and a small convolution kernel obtains a small receptive field, which is beneficial to capture the features of small targets. Many models apply multi-scale feature extraction modules in the deep network to enhance the ability to extract global context feature information, but directly concatenating the feature maps after multiple dilated convolutions does not consider the semantic correlation between the branches, and cannot well handle the diversity of the number and size of salient targets in natural scenes. SUMMARY

[0009] The purpose of the present application is to provide an RGB-D salient target detection method based on boundary deformable convolution guidance to solve the technical problems that the prior art cannot well handle the diversity of the number and size of salient targets in natural scenes, and when multiple targets or small targets are contained, the phenomenon of missed detection occurs, and it is difficult to effectively identify the target structure and contour details, resulting in a blurred and incomplete salient map boundary.

[0010] The RGB-D salient target detection method based on boundary deformable convolution guidance comprises the following steps:

[0011] Step one, respectively extracting features from the RGB modality and the depth map modality;

[0012] Step two, fusing the features of the two modalities through a cross-modal attention fusion feature module to mine the common and complementary features of salient objects;

[0013] Step three, inputting the fused feature map into an encoder deep layer embedded with an adjacent multi-scale feature enhancement module to obtain global context feature information;

[0014] Step four, generating a boundary clue map of salient objects by constructing a boundary feature extraction module;

[0015] Step five, using the generated boundary clue map and deformable convolution guided cross-modal fusion features to generate a salient map layer by layer.

[0016] Preferably, the cross-modal attention enhanced feature module comprises two parts: a cross-modal feature fusion module and an attention mechanism module; after inputting the RGB feature map and the depth feature map, first, the cross-modal feature fusion module is processed, the correlation and complementarity between the two modalities are explored by using the feature cross fusion strategy, the feature maps of the two modalities are cross multiplied, so that the different modal feature maps can mutually mine and strengthen the commonness of salient objects through cross fusion; then, the channel-space attention module is used to process the features after strengthening the commonness of salient objects, and finally the cross-modal fusion features are obtained by concatenating the feature maps of the two modalities.

[0017] Preferably, the adjacent multi-scale feature enhancement module utilizes adjacent hierarchical feature interaction to realize complementary of saliency features, so as to obtain more rich global context feature information; then a channel attention module is utilized to obtain the weight of each branch, so as to achieve the purpose of efficient feature screening, and then all feature maps are spliced as the final output.

[0018] Preferably, the boundary feature extraction module comprises a plurality of boundary decoder blocks, the input of each boundary decoding block is the cross-modal fusion feature map corresponding to the output of the previous boundary decoder block, the two are spliced to obtain a feature map, the obtained feature map is input to a convolution layer, a BN layer and a ReLU activation function for processing, then the feature map is adjusted to a feature map with a channel number of 1 and the same size as the input model picture through bilinear interpolation and convolution, and finally a Sigmoid function is used to obtain the boundary clue map of the salient object; in order to accurately generate the boundary clue map, the RGB-D salient object detection method uses a supervised loss to strengthen the extraction of the target boundary feature by the boundary feature extraction module, and the cross-modal fusion feature is reused through a cascading manner and then input into the next boundary decoder block.

[0019] Preferably, the boundary deformable convolution guiding module is adopted in the step five, the boundary deformable convolution guiding module comprises a boundary guiding module and a deformable convolution module; in the boundary guiding module, the boundary clue map is mapped to a feature map through attention at the feature level, and the feature map is combined with the cross-modal fusion feature map and then input into the deformable convolution module; the deformable convolution module can learn an additional offset, after learning combined with the offset, the position of the deformable convolution kernel can be dynamically adjusted according to the salient object to be detected at present, and the sampling point positions of different position convolution kernels will be adaptively changed according to the position of the salient object; finally, a Sigmoid function is used to generate a saliency map.

[0020] Preferably, in the step one, two completely same parallel backbone networks are adopted to extract the features of the RGB image and the depth image respectively; denoted as F i R and F i D , i represents the layer number of the feature map, i∈{1, 2, 3, 4}, wherein the channel number of the i-th layer is denoted as C i ;

[0021] In the step two, the calculation formula of the cross-modal feature fusion module is as follows:

[0022] F i RS =Cov(Cov(F i R )), (1)

[0023] F iDS = Cov(Cov(F i D )), (2)

[0024]

[0025]

[0026] where Cov denotes convolution operation, denotes element-wise multiplication;

[0027] The calculation formula applied by the channel-space attention module is as follows:

[0028] F i RCS = CA(SA(F i DSD )), (5)

[0029] F i DCS = CA(SA(F i DSR )), (6)

[0030] F i RD = C(F i DCS , F i RCS ), (7)

[0031] where C denotes concatenation operation (Concat), CA denotes channel attention, and SA denotes spatial attention.

[0032] Preferably, in the step three, the adjacent multi-scale feature enhancement module is composed of five branches, and the expansion factor r of each branch is 1, 2, 3, 4 and 5 respectively, and the deep feature map input into the module is F4 RD ; wherein the middle three branches AM2, AM3 and AM4 contain the self branch and the two branches adjacent to it, and the other two branches AM1 and AM5 only contain two branches: one self branch and one adjacent branch, and the specific calculation formula is as follows:

[0033]

[0034] where f d represents dilated convolution, and r represents expansion factor;

[0035] The branch features F i AM The fused branch output F r is obtained by additive fusion.DAM ; the channel attention module CA is used to obtain the weight of each branch, and then all the feature maps are spliced as the final output F AMC ; the calculation formula is as follows:

[0036]

[0037] F AMC = C (F1 CDAM ,..., F r CDAM ), r = 5. (10)

[0038] Preferably, in the fourth step, the boundary feature extraction module comprises four boundary decoder blocks, each boundary decoder block is defined as DeU i , (i ∈ 1,..., 4)), the input of the i-th boundary decoder block DeU i is the output F i+1 of the previous boundary decoder block DeU i+1 DeU and the i-th cross-modal fusion feature map F i RD , and the two are spliced to obtain the feature map F i ERD ; then the feature map F i ERD is input to the convolution layer, BN layer, ReLU activation function, and then adjusted to a feature map with a channel number of 1 and the same size as the input model picture through bilinear interpolation and convolution, and finally the Sigmoid function is used to obtain the boundary clue map Edge i of the salient object; the calculation formula of the above process is as follows:

[0039]

[0040]

[0041] wherein Up represents the bilinear interpolation operation, the subscript ×2 i indicates the up-sampling multiple, f indicates the convolution, BN layer and ReLU activation function, and σ indicates the Sigmoid activation function.

[0042] Preferably, in the fifth step, the generated boundary clue map is used to guide the cross-modal fusion feature map to generate the saliency map step by step; specifically, the boundary clue map is input into the 3 × 3 convolution layer and the Sigmoid activation function, so as to obtain the normalized feature map F i AE , which is multiplied by the cross-modal fusion feature map and then added to output the feature map F i AE .i ARD , the calculation formula is as follows:

[0043] F i AE = σ (Cov (Edg e i)), (14)

[0044]

[0045] wherein indicates that the corresponding elements are added, indicates that the corresponding elements are multiplied, σ indicates a Sigmoid activation function, and Cov indicates a convolution operation;

[0046] The deformable convolution adds a two-dimensional displacement offset to the position of the sampling point in the convolution kernel, and samples each position p0 in the input feature map F i ARD by the following formula:

[0047]

[0048] wherein W() indicates a matrix weight, p n enumerates the positions listed in , Δp n is an offset, so that the position of the sampling becomes irregular, and finally a Sigmoid function is used to generate a saliency map Smap i .

[0049] Preferably, in the training, the RGB-D saliency target detection method optimizes the boundary clue map using a cross-entropy loss function, and the formula of the cross-entropy loss function of the boundary clue map is as follows:

[0050]

[0051] wherein j indicates a pixel value, GT e+ and GT e- respectively indicate a salient edge pixel and a background pixel, L i edge indicates the loss function of the i-th boundary decoding block, W and H respectively indicate the length and width of the feature map;

[0052] In the saliency map branch, the cross-entropy loss function and the IOU loss function are used to optimize the model; the cross-entropy loss function is a BCE loss function, and is expressed as:

[0053]

[0054] wherein P and G indicate a predicted saliency map and a true value map, Px，y and G x，y respectively represent the predicted foreground probability value and the foreground probability value of the ground truth map, L i bce represents the BCE loss function of the i-th layer saliency map branch.

[0055] The IOU loss function is represented as:

[0056]

[0057] where L i IOU represents the IOU loss function of the i-th layer saliency map branch.

[0058] The total loss function of the whole model is:

[0059]

[0060] The present application has the following advantages: the present application uses a feature cross fusion strategy to explore the correlation and complementarity between two modalities, and the feature maps of the two modalities are cross-multiplied, so that different modal feature maps are mined and strengthened by cross fusion. The quality of the depth map, the illumination and background color of the RGB image, etc. have a greater impact on detecting salient objects. The present application uses channel attention to calculate the weight of each channel in the feature map, better measures the importance of each feature channel, and uses spatial attention to measure the importance of different spatial positions, which can better locate the spatial region of the target.

[0061] The present application uses adjacent level feature interaction to realize saliency feature complementation, thereby obtaining more rich global context feature information. This is crucial for capturing salient objects with different sizes and uncertain numbers, thereby overcoming the technical defect that the prior art cannot well handle the diversity of the number and size of salient objects in natural scenes, and better locating the position information of salient objects. When the adjacent multi-scale feature enhancement module of the present method is used, there are more salient features in the branch output of each branch of the module, but at the same time, it is difficult to distinguish the subtle features of salient objects, therefore the present method also uses a channel attention module to obtain the weight of each branch, thereby achieving the purpose of efficient feature screening.

[0062] The prior art has not been able to effectively use the boundary clue map of the saliency map, and in terms of generating a saliency map, the prior art has a boundary blur problem in some complex scenes. The target feature information is gradually converted into boundary feature information by the boundary feature extraction module, thereby obtaining a boundary clue map, and then the generated edge clue map is used to guide the subsequent decoding of the model, which can better solve the aforementioned boundary blur problem. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 The basic flowchart of the RGB-D salient object detection method based on boundary deformable convolution guidance according to the present application.

[0064] Figure 2 The specific flowchart of the detection model according to the present application.

[0065] Figure 3 The schematic diagram of the cross-modal attention enhanced feature module according to the present application.

[0066] Figure 4 The schematic diagram of the adjacent multi-scale feature enhancement module according to the present application.

[0067] Figure 5 The boundary clue map generated by the boundary feature extraction module according to the present application.

[0068] Figure 6 The schematic diagram of the boundary deformable convolution guidance module according to the present application.

[0069] Figure 7 The comparison diagram of the visual saliency maps obtained by the present application and other 12 detection methods.

[0070] Figure 8 The comparison diagram of the P-R and F-measure curves of the present application and other detection methods. DETAILED DESCRIPTION

[0071] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings, and by describing the embodiments, so as to help the skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solutions of the present application.

[0072] As shown in Figures 1-8 The present application provides a RGB-D salient object detection method based on boundary deformable convolution guidance, which first extracts the features of RGB image and depth image by using two completely same parallel backbone networks; then uses a cross-modal attention fusion feature module to mine the commonness of salient objects of RGB and depth images; then, the obtained deep cross-modal feature map is input into an adjacent multi-scale feature enhancement module to obtain context feature information; after that, two branches are generated in the process of decoding to generate a saliency map, and each branch uses an encoder-decoder structure: the first branch is a boundary clue map branch, which generates a boundary clue map of salient objects by using an edge feature extraction module; the second branch is a saliency map branch, and a boundary deformable convolution guidance module is added in this branch, which uses the generated boundary clue map and deformable convolution guided cross-modal fusion features to perform layer-by-layer decoding to generate a clear, complete and accurate saliency map.

[0073] The target detection method specifically comprises the following steps.

[0074] Step one, respectively extracting features of RGB modalities and depth map modalities.

[0075] The image size input into the model is denoted as CxWxH (C, W and H represent the channel number, length and width of the feature map respectively). Taking Swin-Transformer as the feature extractor, two completely same parallel backbone networks are used to extract RGB image features and depth image features respectively, denoted as F i R and F i D , i∈{1, 2, 3, 4}, i represents the layer number of the feature map, the first layer feature map is scaled to W / 4xH / 4, and the resolution of the feature map of the last three layers is scaled to W / 2 i+1 xH / 2 i+1 , i∈{2, 3, 4}, wherein the channel number of the i-th layer is denoted as C i , C i ∈{128, 256, 512, 1024}.

[0076] Step two, fusing the features of the two modalities through the cross-modal attention fusion feature module to mine the commonness and complementary features of the salient objects.

[0077] The cross-modal attention enhanced feature module (Cross-modal attention feature fusion module, CMAF) adopted by the application is as shown in Figure 2 , which comprises two parts: a cross-modal feature fusion module and an attention mechanism module.

[0078] Firstly, the RGB feature map and the depth feature map are respectively input into the 2-layer 3x3 convolution layer through the cross-modal feature fusion module, and then the feature cross fusion strategy is used to explore the correlation and complementarity between the two modalities, the feature maps of the two modalities are cross-multiplied, so that the different modal feature maps cross-fuse to mutually mine and strengthen the commonness of the salient objects. The calculation formula of the cross-modal feature fusion module is as follows:

[0079] F i RS =Cov(Cov(F i R )), (1)

[0080] F i DS =Cov(Cov(F i D )), (2)

[0081]

[0082]

[0083] where Cov denotes a convolution operation, denotes element-wise multiplication.

[0084] After that, the channel-space attention module is used to process the features after the commonness of the reinforced salient objects, and finally the cross-modal fusion features F are obtained by cascading the two modal feature maps RD The quality of the depth map, the illumination of the RGB image and the background color have a great influence on the detection of salient objects. The channel-space attention module is used to solve the above problems. The channel attention calculates the weight of each channel in the feature map, which better measures the importance of each feature channel. The spatial attention measures the importance of different spatial positions, which can better locate the spatial region of the target. The calculation formula of the channel-space attention module is as follows:

[0085] F i RCS =CA(SA(F i DSD )), (5)

[0086] F i DCS =CA(SA(F i DSR )), (6)

[0087] F i RD =C(F i DCS ,F i RCS ), (7)

[0088] where C denotes a concatenation operation (Concat), CA denotes channel attention, and SA denotes spatial attention.

[0089] Step three, input the fused feature map into the deep layer of the encoder embedded with the adjacent multi-scale feature enhancement module to obtain global context feature information.

[0090] The deep feature map contains rich salient target position information, and a large convolution kernel can obtain a large receptive field, which is conducive to capturing the features of large targets, while a small convolution kernel obtains a small receptive field, which is conducive to capturing the features of small targets. Many models apply multi-scale feature extraction modules in the deep layer of the network to enhance the extraction ability of global context feature information, but directly cascading the feature maps after multiple dilated convolutions does not consider the semantic correlation between branches, and cannot well process the diversity of the number and size of salient targets in natural scenes.

[0091] The present application designs an adjacent multi-scale feature enhancement module (AMFE) to realize complementary significant feature by adjacent level feature interaction, so as to obtain more abundant global context feature information. This is crucial for capturing significant objects with different sizes and uncertain number, thus overcoming the defects of the prior art, and better positioning the position information of the significant object. The detailed structure of the adjacent multi-scale feature enhancement module is shown in Figure 4 .

[0092] Specifically, the adjacent multi-scale feature enhancement module is composed of five branches, and the expansion factor r of each branch is 1, 2, 3, 4 and 5 respectively. The deep feature map input into the module is F4 RD . Among them, the middle three branches AM2, AM3 and AM4 contain their own branches and two branches adjacent to them, while the other two branches AM1 and AM5 only contain two branches: one own branch and one adjacent branch. The specific calculation formula is as follows:

[0093]

[0094] Where f d represents dilated convolution, and r represents expansion factor.

[0095] The branch feature F r AM After addition fusion, the fused branch output F r DAM is obtained. This makes the branch output F r DAM of each branch AM exist more significant features. But at the same time, it will cause the difficulty in distinguishing the subtle features of the significant target, so the channel attention module CA is used to obtain the weight of each branch, so as to achieve the purpose of efficient feature screening, and then all the feature maps are spliced as the final output F AMC . The calculation formula is as follows:

[0096]

[0097] F AMC =C(F1 CDAM ,...,F r CDAM ), r = 5. (10)

[0098] Step four, generate the boundary line clue map of the significant object by constructing the boundary feature extraction module.

[0099] The prior art has not been able to effectively utilize the boundary clue map of the saliency map, and the prior art has a boundary blur problem in generating the saliency map in some complex scenes. The target feature information is gradually converted into boundary feature information by the boundary feature extraction module, so as to obtain the boundary clue map, and then the generated edge clue map is used to guide the subsequent decoding of the model, so that the foregoing boundary blur problem can be better solved.

[0100] Specifically, the boundary feature extraction module (BFEM) includes 4 boundary decoder blocks, each boundary decoding block is defined as DeU i , (i∈1,..., 4)), for the i-th boundary decoder block DeU i , the input is the output F i+1 of the previous boundary decoder block DeU i+1 DeU and the i-th cross-modal fusion feature map F i RD The two are spliced to obtain the feature map F i ERD . Then the feature map F i ERD is input to the convolution layer, the BN layer, the ReLU activation function, then the feature map is adjusted to a feature map with a channel number of 1 and the same size as the input model picture through bilinear interpolation and convolution, and finally the Sigmoid function is used to obtain the boundary clue map Edge i of the salient object. In addition, in order to accurately generate the boundary clue map Edge i , the method uses a supervised loss to strengthen the extraction of the target boundary feature by the boundary feature extraction module. In order to ensure the integrity of the edge structure of the salient object, the cross-modal fusion features are reused in a cascading manner and then input to the next boundary decoder block. The generated boundary clue map is as shown in Figure 5 The calculation formula of the above process is as follows:

[0101]

[0102]

[0103] Wherein, Up represents the bilinear interpolation operation, the subscript ×2 i represents the up-sampling multiple, f represents the convolution, BN layer and ReLU activation function, and σ represents the Sigmoid activation function.

[0104] In the training, the cross-entropy loss function is used to optimize the boundary clue map, and the formula of the cross-entropy loss function of the boundary clue map is as follows:

[0105]

[0106] where j denotes the pixel value, GT e+ and GT e- denote the salient edge pixel and background pixel respectively, L i edge denotes the i-th boundary decoded block loss function, W and H denote the length and width of the feature map respectively.

[0107] Step five, using the generated boundary cue map and deformable convolution to guide the cross-modal fusion feature to generate saliency map layer by layer.

[0108] Generally, the boundary contour of many targets in natural images is complex and changeable, and it is difficult to accurately extract boundary features by using conventional convolution operation, because the conventional convolution is limited by fixed geometric structure when constructing model transformation. The limitation determines that the convolution unit can only sample the input image at a fixed position, which leads to weak representation ability of the extracted features. Therefore, the present application proposes a boundary deformable convolution guidance module (BDCG), which can effectively extract the complex boundary features of the target, thereby improving the modeling ability of deformation. The structure of the boundary deformable convolution guidance module is as shown in Figure 6 .

[0109] The boundary feature information is helpful to generate a saliency map with fine boundaries. The present method uses the generated boundary cue map to guide the cross-modal fusion feature map to generate a saliency map step by step (boundary guidance module). Specifically, the boundary cue map is input into a 3x3 convolution layer and a Sigmoid activation function, so as to obtain a normalized feature map F i AE , which can be regarded as a feature-level attention map. The feature map F i AE is multiplied by the cross-modal fusion feature map and then added to output a feature map F i ARD , which can adaptively improve the feature representation ability of the salient object. The calculation formula of the above process is as follows:

[0110] F i AE =σ(Cov(Edge i )), (14)

[0111]

[0112] where denotes the addition of corresponding elements, denotes multiplication of corresponding elements, denotes a sigmoid activation function, and Cov denotes a convolution operation.

[0113] The feature map F obtained by combining the above cross-modal fusion feature map is input into a deformable convolution module. i ARD The input is input into a deformable convolution module. The deformable convolution module can learn an additional offset. After learning the offset, the position of the deformable convolution kernel can be dynamically adjusted according to the salient object to be detected at present. The sampling point position of the convolution kernel at different positions will be adaptively changed according to the position of the salient object, so as to better fit the geometric deformation characteristics of the salient object in a natural scene.

[0114] The regular convolution is used to sample each position p0 in the input feature map F i ARD Each position p0 in the input feature map F

[0115]

[0116] where W() denotes a matrix weight, p n The positions listed in the input feature map F are enumerated,

[0117] The deformable convolution adds a two-dimensional displacement offset to the position of the sampling point in the convolution kernel. The offset is learned by the convolution layer from the input feature map. The offset and the input feature map are input into a subsequent convolution layer, as shown in FIG. 4. Figure 6 The same sampling position p0 becomes

[0118]

[0119] where Δp n is an offset (usually a decimal number), so that the position of the sampling becomes an irregular position. Finally, a sigmoid function is used to generate a saliency map Smap i The embodiment uses four saliency map decoder blocks. The deformable convolution is used in the second, third and fourth saliency map decoder blocks, and the deformable convolution is not used in the first saliency map decoder block. Only the boundary guided module is used to generate the saliency map.

[0120] The boundary deformable convolution guided module used in this step can fuse the saliency region features and the boundary features, use the boundary clue map to shield the non-saliency region, and use the deformable convolution to extract the irregular position features of the salient object to improve the performance of the detection model of the method in a complex scene.

[0121] In the saliency map branch, the model is optimized using a cross-entropy loss function and an IOU loss function. The cross-entropy loss function is a BCE loss function, which is represented as:

[0122]

[0123] where P and G represent the predicted saliency map and ground truth map, P x，y and G x，y represent the predicted foreground probability value and the foreground probability value of the ground truth map, respectively, L i bce represents the BCE loss function of the i-th layer saliency branch.

[0124] The IOU loss function is represented as:

[0125]

[0126] where L i IOU represents the IOU loss function of the i-th layer saliency branch.

[0127] The total loss function of the entire model is:

[0128]

[0129] The actual effect of the above scheme is described below in combination with specific experiments.

[0130] The present application is compared extensively in six public benchmark RGB-D datasets, which are LFSD, NLPR, NJU2K, SSD, SIP and STERE. The LFSD dataset is collected by Lytro camera, which includes 100 pairs of pictures, and the salient region is determined by 3 people together. NLPR includes 1000 image pairs with a resolution of 640*480, and multiple salient objects exist in some images. The depth map is obtained by Microsoft Kinect under different lighting conditions. NJU2K is the largest RGB-D dataset, which contains 1985 image pairs. SSD selects 80 images from three stereo movies of indoor and outdoor scenes to form a test set, and the resolution of each image is as high as 960*1080, but the depth map quality is relatively rough. SIP uses 1000 image pairs captured by Huawei dual-camera smart phone, and the image size resolution is 992*744. The image contains real scenes from various perspectives, poses, occlusions, low light and complex backgrounds. STERE is the first stereo image dataset, which contains 1000 pairs of binocular images. These images are mainly collected from the Internet.

[0131] As Figure 7As shown in the figure, the saliency map obtained by the method is compared with the visual saliency maps obtained by other 12 methods. As can be seen from the first, second, third, fourth, fifth, sixth and seventh rows of images, the boundary profile of the salient object is complex, and for the object in the fifth row, many models not only predict the salient object boundary inaccurately, but also predict the branches as the object, in comparison, since the boundary saliency map is introduced into the model by the method, the generated saliency map has clear and complete boundary profile. As can be seen from the tenth and eleventh rows, the color of the salient object and the background and their similarity, in the eleventh row, the salient object foot is lost in the saliency map generated by many methods, although the VST method predicts the salient object foot, but the boundary profile is also relatively fuzzy, while the saliency map predicted by the method has complete and clear boundary. In addition, thanks to the effect of the adjacent multi-scale feature enhancement module, the method also has good detection results on large targets (eighth and ninth rows), small targets (tenth and eleventh rows) and multiple targets (last row). In summary, the visual saliency map of the method is significantly better than other models, and closer to the true value map than other models.

[0132] As Figure 8 shown, the P-R curve and F-measure curve of the method and other methods are compared. As can be seen, the area enclosed by the P-R curve and the coordinate axis of the method is larger, indicating that the detection effect of the method is better, and the curve is shorter, indicating that the recall rate is higher. As can be seen from the F-measure curve, the curve of the method is obviously higher than other saliency detection methods on the LFSD, NLPR, SSD, STERE and NJU2K data sets, and is also higher than most detection methods on the SIP data set.

[0133] The above describes the present application in conjunction with the drawings, and it is obvious that the specific implementation of the present application is not limited by the above method, as long as various non-essential improvements are made by using the inventive concept and technical solution of the present application, or the inventive concept and technical solution of the present application is directly applied to other occasions without improvement, all within the scope of protection of the present application.

Claims

1. A method for RGB-D salient object detection based on boundary deformable convolution guidance, characterized in that: Comprise the following steps: Step one, respectively, to RGB modalities and depth map modalities for feature extraction; Step two, through the cross-modal attention fusion feature module to two kinds of modalities feature fusion to excavate the commonness and complementary features of the salient object; Step three, the feature map obtained by fusion is input into the encoder deep layer embedded with adjacent multi-scale feature enhancement module to obtain global context feature information; Step four, the boundary clue map of the salient object is generated by constructing the boundary feature extraction module; Step five, the generated boundary clue map and deformable convolution guide cross-modal fusion features are used to generate saliency map layer by layer; In the step three, the adjacent multi-scale feature enhancement module is composed of five branches, each branch adopts an expansion factor r of 1, 2, 3, 4, 5 respectively, and the deep feature map input to the module is F 4 RD ; wherein, the middle three branches AM2, AM3 and AM4 contain the self branch and the two branches adjacent to it, and the other two branches AM1 and AM5 only contain two branches: one self branch and one adjacent branch, and the specific calculation formula is as follows: ，（8） wherein f d denotes a dilated convolution, r denotes a dilation factor; The included branch feature F r AM The fused branch output obtained by additive fusion F r DAM The channel attention module is utilized CA to obtain the weight of each branch, and then all the feature maps are spliced as the final output F AMC The calculation formula is as follows: ，（9） ；（10） The boundary deformable convolution guiding module in the fifth step includes a boundary guiding module and a deformable convolution module; in the boundary guiding module, the boundary cue map is mapped to a feature map through attention of a feature level, and the feature map is combined with the cross-modal fusion feature map and then input into the deformable convolution module; the deformable convolution module can learn an additional offset, and after learning combined with the offset, the position of the deformable convolution kernel can be dynamically adjusted according to a significant object to be detected currently, and the sampling point positions of different position convolution kernels will be adaptively changed according to the position of the significant object; finally, a saliency map is generated by using Sigmoid a function. 2.The method of claim 1, wherein the method is based on boundary deformation convolution guidance for RGB-D salient object detection. The cross-modal attention fusion feature module includes two parts: cross-modal feature fusion module and attention mechanism module; after inputting the RGB feature map and the depth feature map, first, the cross-modal feature fusion module is processed, the feature cross fusion strategy is used to explore the correlation and complementarity between the two modalities, the feature maps of the two modalities are cross multiplied, so that the different modal feature maps can mutually excavate and strengthen the commonness of the salient object; then the channel-space attention module is used to process the features after strengthening the commonness of the salient object, and finally the cross-modal fusion features are obtained by cascading the two modal feature maps. 3.The method of claim 1, wherein the method further comprises: The adjacent multi-scale feature enhancement module uses adjacent layer feature interaction to realize saliency feature complementarity, so as to obtain more rich global context feature information; Then, the channel attention module is used to obtain the weight of each branch, so as to achieve the purpose of efficient feature screening, and then all the feature maps are spliced as the final output. 4.The method of claim 1, wherein the method further comprises: The boundary feature extraction module comprises a plurality of boundary decoder blocks, the input of each boundary decoder block is the cross-modal fusion feature map corresponding to the output of the previous boundary decoder block, the two are spliced to obtain a feature map, the obtained feature map is input into a convolution layer, BN layer, ReLU an activation function for processing, then the feature map is adjusted to a feature map with a channel number of 1 and the same size as the input model picture through bilinear interpolation and convolution, and finally a boundary clue map of a salient object is obtained through a Sigmoid function; in order to accurately generate the boundary clue map, the RGB-D salient object detection method uses a supervised loss to strengthen the extraction of the target boundary feature by the boundary feature extraction module, and the cross-modal fusion feature is reused in a cascading manner and then input into the next boundary decoder block.

5. The method of claim 2, wherein the method is a boundary deformation convolution guided RGB-D salient object detection method. In the step one, two completely same parallel backbone networks are used to extract RGB image and depth map features; denoted as F i R and F i D , i denotes the number of feature map layers, i ∈{1, 2, 3, 4}, where the first i layer has a number of channels denoted as C i ; In the step two, the calculation formula of the cross-modal feature fusion module is as follows: ，（1） ，（2） ，（3） ， （4） wherein Cov denotes a convolution operation, denotes multiplication of corresponding elements; The calculation formula applied by the channel-space attention module is as follows: ，（5） ，（6） ，（7） wherein C represents a concatenation operation Concat ), CA represents a channel attention, SA represents a spatial attention.

6. The method of claim 5, wherein the method is a boundary deformation convolution guided RGB-D salient object detection method. In the fourth step, the boundary feature extraction module includes four boundary decoder blocks, each of which is defined as DeU i , i ∈{1, 2, 3, 4}, for the first i boundary decoder block DeU DeU i , the input is the output of the previous boundary decoder block DeU i+1 F i+1 DeU , and the first i cross-modal fusion feature map F i RD , and the two are spliced to obtain a feature map F i ERD ; then the feature map F i ERD is input to the convolution layer, BN layer, ReLU activation function, and then the feature map is adjusted to a feature map with a channel number of 1 and the same size as the input model picture through bilinear interpolation and convolution, and finally the boundary clue map of the salient object is obtained through the Sigmoid function Edge i ; the calculation formula of the above process is as follows:​ ，（11） ，（12） wherein, Up denotes a bilinear interpolation operation, the subscript x2 i denotes a factor of upsampling, f denotes a convolution, BN layer and ReLU activation function, denotes Sigmoid activation function.

7. The method of claim 6, wherein the method is a boundary deformation convolution guided RGB-D salient object detection method. In the step five, the generated boundary clue map is used to guide the cross-modal fusion feature map to generate saliency map step by step; Specifically, the boundary cue graph is input into a 3×3 convolutional layer and Sigmoid In the activation function, a normalized feature map is obtained. F i AE , feature map F i AE Multiply and then add the cross-modal fusion feature map to output the feature map. F i ARD The calculation formula is as follows: ，（14） ，（15） wherein denotes addition of the corresponding elements, denotes multiplication of the corresponding elements, denotes Sigmoid activation function, Cov denotes convolution operation; Deformable convolution adds a two-dimensional displacement offset to the position of the sampling points in the convolution kernel, for each input feature map F i ARD each position in the input feature map p 0 is calculated by the following equation: ，（17） wherein, W ( ) denotes a matrix weight, p n enumerated the positions listed in = {(-1,-1), {-1,0},… (0,1), (1,1)}; Δ p n is an offset, so that the position of the sampling becomes an irregular position, and finally a Sigmoid function is used to generate a saliency map Smap i .

8. The method of claim 7, wherein the method is a boundary deformation convolution guided RGB-D salient object detection method. In the training, the RGB-D saliency target detection method uses cross entropy loss function to optimize the boundary clue map, and the calculation formula of the cross entropy loss function of the boundary clue map is as follows: ，（13） wherein j represents a pixel value, GT e+ and GT e- respectively represent a salient edge pixel and a background pixel, L i edge represents the i-th boundary-decoding block loss function, i W and H respectively represent the length and width of the feature map;​ In the saliency map branch, the cross entropy loss function and the IOU loss function are used to optimize the model; the cross entropy loss function is the BCE loss function, which is expressed as: ，（18） wherein P and G denote the predicted saliency map and the ground truth map, P x,y and G x,y denote the predicted foreground probability value and the ground truth foreground probability value, respectively, L i bce denote the BCE loss function of the first i layer saliency map branch; The IOU loss function is expressed as: ，（19） wherein L i IOU representing the i IOU loss function of the layer saliency branch Then the total loss function of the whole model is: （20）。

Images