RGB-d saliency detection method based on progressive multi-scale feature fusion

Abstract

The RGB-D saliency detection method based on progressive multi-scale feature fusion comprises the following steps: 1) obtaining an RGB-D dataset for training and testing the task; 2) selecting a double-flow Swin-Transformer model as a backbone network to extract RGB features and depth features in four stages during model training; 3) obtaining cross-modal fusion features in the four stages through a cross-attention fusion module; 4) using a spatial progressive fusion module to aggregate the cross-modal fusion features in the four stages into final decoding features; through three weight fusion modules, the cross-modal fusion features in the first three stages are sequentially fused into the decoding features; 5) after each weight fusion module, an progressive convolution module is used to extract features from the feature map after weight fusion; and the output feature map of the last progressive convolution module is used as the result of RGB-D saliency target detection. The self-attention mechanism of the application can directly model the relationship between different positions on the feature map without distance restriction, so that long-distance dependencies in the image can be effectively captured.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, more particularly to an RGB-D saliency detection method based on progressive multi-scale feature fusion BACKGROUND

[0002] RGB-D salient object detection (SOD) is a computer vision task aiming to achieve accurate segmentation of salient objects by analyzing the complementary information of RGB images and depth images. Currently, most RGB-D SOD models adopt single-stream or dual-stream structure in the design of backbone network, among which the dual-stream structure is widely adopted due to its advantage in cross-modal information capture. Existing models usually extract image features based on convolutional neural networks (CNN), achieving good detection results. However, the feature extraction method of CNN based on sliding window has inherent limitations, making it difficult to effectively capture global features in the image, resulting in insufficient modeling ability for long-distance context dependency, thus limiting the detection performance.

[0003] With the introduction of Vision Transformer (ViT), its strong global modeling ability provides a new approach to salient object detection. Some researchers have tried to replace the traditional CNN backbone with a Transformer-based backbone, further improving the detection results. However, how to effectively fuse the complementary information of RGB features and depth features remains one of the core challenges in the field of RGB-D SOD. Existing methods mostly achieve cross-modal feature fusion through multi-path fusion, cross-level interaction, and additional feature enhancement modules, but these methods generally have high computational overhead, feature redundancy, and insufficient long-range cross-modal feature interaction.

[0004] On the other hand, to adapt to the diversity of target size and environment in the image, multi-scale feature fusion technology has become one of the key technologies to improve SOD performance. Existing research usually enhances multi-scale features through attention mechanisms, spatial pyramid pooling (ASPP), or context interaction modules. However, in the multi-scale feature fusion stage, direct concatenation of features or multiple sampling operations may lead to feature redundancy, noise introduction, and information loss, thereby affecting the prediction accuracy of salient objects.

[0005] In summary, existing RGB-D SOD methods have limitations in cross-modal fusion and multi-scale feature fusion, and there is an urgent need for a new method that can efficiently extract and fuse multi-modal and multi-scale information to improve the accuracy and robustness of salient object detection. SUMMARY

[0006] In view of the existing problems, in order to solve the problems existing in the prior art, the technical field of the problems is analyzed, and an RGB-D saliency detection method based on progressive multi-scale feature fusion is proposed.

[0007] The technical scheme of the present application is as follows:

[0008] The RGB-D saliency detection method based on progressive multi-scale feature fusion comprises the following steps:

[0009] Step 1, obtaining the training set and test set required by the task.

[0010] Step 2, using a double-flow Swin-transformer network as the backbone network to extract different scale RGB image features and depth image features in the four stages of the model training process, respectively denoted as f i r and f i d , i∈(1,4).

[0011] Step 3, using a cross attention fusion module (CAFM, Cross Attention Fusion Module) to respectively fuse the output features f i r / d and in the adjacent layers of the RGB branch and the depth branch into F i r / d , i∈(1,4). Then, cross-modal feature fusion is performed on and to obtain multi-level cross-modal fusion features

[0012] Step 4, using a spatial progressive fusion module (SPFM, Spatial Progressive Fusion Module) to progressively extract low-level spatial information and integrate it into high-level features in the order from low-level features to high-level features according to the cross-modal fusion features obtained in step 3, to obtain the final RGB-D image encoding features

[0013] Step 5, through upsampling and channel compression operations, the encoding features obtained in step 4 are aligned with the feature dimensions of the stage cross-modal features obtained in step 3, to obtain From the third stage, the weight fusion module (WFM, Weight Fusion Module) is used to and The weighted splicing on the channel is performed, the splicing result is decoded by using a difference convolution module (PCM, Progressive Convolution Module) to obtain The final is obtained as a result of saliency prediction.

[0014] Further, 1485 groups, 700 groups, and 800 groups of images are selected from the public data sets NJU2K, NLPR, and DUT respectively as a training set, each group of images including three pictures of RGB images, depth images, and saliency real images. 135 groups, 400 groups, 100 groups, 500 groups, 300 groups, and 929 groups of images are selected from the public data sets DES, DUT, LFSD, NJU2K, NLPR, and SIP respectively as comparative test sets. In the implementation, the training and test data sets can be replaced according to specific application objects.

[0015] Further, in step 2, a symmetric double-flow Swin-Transformer of the Swin-B version is used as the backbone network to process the RGB image and the depth image respectively. The different scale RGB image features and depth image features in the four stages of the model training process are extracted, and are denoted as f i r and f i d .

[0016] Further, in step 3, the cross-attention fusion module is used to upsample the high-level feature to restore the pixels, and then splice (the fourth layer is spliced with itself) with the low-level feature f i r / d to fuse the adjacent f i r / d and f into F i r / d . Then, the spatial coordinate attention is calculated once, the coordinate attention map obtained is multiplied by F i r / d to obtain the coordinate attention feature map. Then, the spatial attention is calculated once, the spatial attention map obtained is multiplied by F i r / d to obtain the multi-level cross-modal fusion feature F i r / d .

[0017] Further, in step 4, the spatial progressive fusion module is used to start from the first stage of the backbone network, and sequentially fuse the cross-modal fusion features F After downsampling, the spatial attention is calculated once and its spatial attention map is integrated into the adjacent high-level cross-modal fusion features. In this way, the global features of different scales are gradually aggregated to obtain the final RGB-D image encoding features.

[0018] Furthermore, in step 5, the coding features obtained in step 3 are first Upsampling and convolution will The pixels and channels are restored to the same characteristic shape as the third layer features, and we get Then use the weight fusion module to combine the cross-modal fusion features of the third stage obtained in step 3 After the channel weights are calculated by channel attention, the weighted splicing is performed again. The weighted splicing result is then fused with features through the residual convolution module to obtain Using three sets of paired weight fusion modules and progressive convolution modules, Integration get Will eventually get This is taken as the result of the significant prediction.

[0019] The second aspect of the present invention relates to a rail surface defect detection method that applies the RGB-D saliency detection method based on progressive multi-scale feature fusion of the present invention, using the industrial RGB-D dataset NEU-RSDDSAUG as a task dataset to construct and train a model; and using the trained model to perform defect detection on collected rail surface images.

[0020] The beneficial effects of the present invention are as follows:

[0021] 1. This paper uses the Swin-Transformer network as the backbone network. Its inherent self-attention mechanism can directly model the relationship between different positions on the feature map without being restricted by distance, thereby effectively capturing long-range dependencies in the image.

[0022] 2. The design of CAFM adopts the tandem use of coordinate attention and spatial attention, which effectively focuses on and mines the long-distance dependency between the target and the background in the cross-modal features, and obtains a wider range of attention information without adding additional feature enhancement modules, thereby achieving full cross-modal fusion of RGB features and depth features.

[0023] 3. In the design of CAFM, SPFM, WFM, RCM and other modules, different multi-scale fusion strategies are adopted before and after cross-modal fusion and in the decoding stage. The features are condensed and extracted from space and channel in turn. This method fully fuses global context information and improves the detection accuracy of objects of different scales. At the same time, in the encoding stage and the decoding stage, the gradual weighting strategy also effectively alleviates the problems of feature loss and redundancy in the feature sampling and splicing process. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is the flow framework diagram of the present application.

[0025] Figure 2 is the network structure schematic diagram of the present application.

[0026] Figure 3 is the channel cross-attention fusion module (CAFM) structure diagram.

[0027] Figure 4 is the space progressive fusion module (SPFM) structure diagram.

[0028] Figure 5 is the weight fusion module (WFM) structure diagram.

[0029] Figure 6 is the progressive convolution module (PCM) structure diagram. DETAILED DESCRIPTION

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0031] The present embodiment provides a steel rail surface defect detection method based on RGB-D saliency detection method of progressive multi-scale feature fusion.

[0032] As a key infrastructure for railway transportation, the surface defects of the steel rail (such as holes, grooves, scars, protrusions, etc.) are directly related to the safety and efficiency of train operation. Therefore, steel rail defect detection has become a crucial task in railway maintenance. Traditional steel rail detection methods mainly rely on manual inspection or single-modal computer vision algorithms, although some results have been achieved, but when facing complex background interference, small defect positioning and simultaneous detection of multiple types of defects, there are still problems of insufficient precision and poor robustness.

[0033] To accommodate the diverse size, shape, and distribution of rail defects, multi-scale feature fusion has become a key technology for improving detection performance. Existing research typically enhances multi-scale features through attention mechanisms, spatial pyramid pooling (ASPP), or contextual interaction modules. However, during the multi-scale feature fusion stage, directly concatenating features or performing multiple sampling operations can lead to feature redundancy, noise introduction, and information loss, which in turn affects the detection accuracy and localization of defect areas.

[0034] In summary, existing RGB-D SOD methods have limitations in cross-modal and multi-scale feature fusion. To meet the specific needs of rail defect detection, a new method is urgently needed that can efficiently extract and fuse multi-modal and multi-scale information to improve the accuracy and robustness of defect detection and provide strong support for railway transportation safety.

[0035] like Figure 1 As shown, the method of this embodiment includes the following specific steps:

[0036] Step 1: Use the NEU-RSDDSAUG industrial RGB-D dataset as the task dataset. Use MATLAB software to segment the images using a random grid for data augmentation. After filtering out low-quality images, 1862 image pairs were generated. Use the LabelImg annotation tool to annotate four defect types: holes, grooves, scars, and bumps. Then, use a ratio of 8:2 to create a training set of 1490 image pairs and a test set of 372 image pairs.

[0037] Step 2: Figure 2 As shown in the figure, the symmetrical two-stream Swin-Transformer of Swin-B version is used as the backbone network to process RGB images and depth images respectively. The RGB image features and depth image features of different scales at four stages in the model training process are extracted and recorded as f i r and f i d ,i∈(1,4).

[0038] Step 3: Figure 3 As shown in Figure 2, the cross attention fusion module mainly includes the calculation of coordinate attention, the calculation of spatial attention, and several splicing and convolution operations. The function of the cross attention fusion module is to combine the high-level features of the RGB stream and the depth stream of each stage. Upsampling restores its pixels and then combines them with the low-level features f i r / d Perform splicing (splice the highest layer with itself), and obtain the adjacent layer splicing feature F through convolution operation i r / di∈(1,4), the preliminary stage splicing features F are obtained through splicing operation i i∈(1,4). Then, F i The coordinate attention is calculated, and the specific operation is to multiply F i r / d One-dimensional horizontal pooling and one-dimensional vertical pooling are performed. The pooled features F i h / w i∈(1,4) are spliced, and global features are extracted through convolution. Then, the horizontal and vertical attention weight matrices are calculated by dividing the global features, and the two matrices are multiplied to obtain the coordinate attention map, which is then multiplied with F i to obtain the coordinate attention feature F Then, the spatial attention is calculated, and the specific operation is to perform global maximum pooling on the feature map, extract features through Con7x7 convolution operation, extract the spatial attention map through sigmoid activation function, and finally multiply F i r / d to obtain the multi-level cross-modal fusion feature F

[0039] The formula of this step is as follows:

[0040]

[0041] Where ConvBR(·) is the cascade operation of convolution, normalization and ReLu function, Up(·) is the up-sampling operation, Cat(·) is the splicing operation, SA(·) is the spatial attention calculation operation, CDA h / w (·) is the horizontal and vertical coordinate attention map generation operation, ConvBS(·) is the convolution, normalization and sigmoid activation function cascade operation, is the element-wise multiplication operation.

[0042] Step 4, if Figure 4 As shown in the figure, the spatial progressive fusion module mainly includes the calculation of spatial attention, down-sampling and residual connection operations. The specific implementation is as follows: the cross-modal fusion features F of the low layer are down-sampled to calculate the spatial attention. The specific operation of spatial attention calculation is to perform global maximum pooling on the feature map, extract features through convolution operation, and extract the spatial attention map through sigmoid activation function. The spatial attention map extracted from is fused into F by residual connection. Three spatial progressive fusion modules are used in series to realize the aggregation of four-stage cross-modal fusion features, and the final RGB-D image encoding feature F

[0043] The formula of this step is as follows:

[0044]

[0045] where SA(·) is a spatial attention computation operation, Down(·) is a down-sampling operation, is an element-wise multiplication operation, is an element-wise addition operation.

[0046] Step 5, as shown in Figure 5 , the weight fusion module mainly includes the calculation of channel attention weight and a number of splicing, convolution, normalization and activation operations. The specific implementation process is: the encoded features obtained in step 3 are up-sampled and once convolved to restore their pixel and channel numbers to the same feature shape as , obtaining Then use the weight fusion module to splice and Through the channel attention module, the channel weight p i is calculated. The specific operation of channel weight calculation is: the input features are once globally averaged and pooled, the dimension is expanded through a linear mapping layer, the data is arranged through a relu activation function, and the dimension is restored through a linear mapping. The channel weight is extracted using a sigmoid activation function. After obtaining the channel weight, the channel weight is used to splice and to obtain As shown in Figure 6 , the progressive convolution module mainly includes a depth separable convolution and two point convolution operations. The specific implementation process is: for the weight splicing feature map obtained by the weight fusion module, the channel number is compressed through convolution, adjusted through batch normalization and GeLu function, and then up-sampled to restore the pixel size. The depth separable convolution is used to extract the information of each channel in , and then the channel expansion and channel compression two point convolutions are used to extract the overall channel features, and then is obtained. Using three pairs of weight fusion modules and progressive convolution modules, in turn is integrated into to obtain The final is obtained, which is used as the result image of rail defect detection. In the rail defect detection task, the trained model is used to process and judge the real-time collected rail surface image. According to the detection result image of the model, it is judged whether there is a defect on the rail surface and the shape and position of the defect.

[0047] The formula of this step is as follows:

[0048]

[0049] wherein CA(·) is a calculation channel attention operation, Down(·) is a down-sampling operation, ConvBR(·) is a joint operation of convolution plus batch normalization plus ReLu activation function, BR(·) is a joint operation of batch normalization plus ReLu activation function, Sigmoid(·) is a Sigmoid activation function, p i is a channel weight, is a feature after fusing the channel weights, is an element-wise addition operation, and x is a matrix multiplication operation. ConvBG(·) is a joint operation of product plus batch normalization plus GeLu activation function, DwConv(·) is a deep separable convolution operation, Pw1Conv(·) is an expansion point convolution operation, and Pw2Conv(·) is a compression point convolution operation.

[0050] Step 6, using a depth camera to inspect the steel rail, collecting the image information of the rail surface in real time, using the trained model to process and judge the collected image, and identifying whether there is a defect in the result.

[0051] The embodiments of the present specification are merely enumerations of implementation forms of the inventive concept, and are only for the purpose of description. The protection scope of the present application should not be regarded as being limited to the specific forms presented in the present embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A method for RGB-D saliency detection based on progressive multi-scale feature fusion, characterized in that: Comprise the following steps: Step 1, obtain the training set and test set required for the task; Step 2, a double-flow Swin-transformer network is used as the backbone network to extract the features of RGB images and depth images of different scales in the four stages of the model training process, respectively denoted as and f i d ,i∈(1,4); Step 3, the output features f i r / d and are fused into F i r / d , i∈(1,4); and the and are cross-modal feature fused to obtain multi-level cross-modal fusion features Step 4, using a spatial progressive fusion module (SPFM), the cross-modal fusion features obtained in step 3 are fused to obtain the final RGB-D image coding features The low-level spatial information is progressively extracted and fused into the high-level features in order from low-level features to high-level features to obtain the final RGB-D image coding features Step 5: Through upsampling and channel compression operations, the encoded features obtained in step 4 are converted to and the cross-modal features obtained in step 3 The feature dimensions of Starting from the third stage, the weight fusion module (WFM) is used to and Perform weighted splicing on the channel and use the difference convolution module (PCM) to decode the splicing results to obtain The final As a result of significant prediction. 2.The RGB-D saliency detection method based on progressive multi-scale feature fusion according to claim 1, characterized in that, The specific method of step 1 is: 1485 groups, 700 groups and 800 groups of images are selected from the public data sets NJU2K, NLPR and DUT respectively as the training set, each group of images includes three pictures of RGB image, depth image and saliency real image;135 groups, 400 groups, 100 groups, 500 groups, 300 groups and 929 groups of images are selected from the public data sets DES, DUT, LFSD, NJU2K, NLPR and SIP respectively as the contrast test set;In the implementation, the training and test data sets can be replaced according to specific application objects. 3.The method of claim 1, wherein, The specific method of step 2 is: using the symmetric double-flow Swin-Transformer of Swin-B version as the backbone network to process the RGB image and the depth image respectively; extracting the RGB image features and the depth image features of different scales in four stages of the model training process, and respectively denoted as and f i d i∈(1,4). 4.The method of claim 1, wherein, The cross attention fusion module in step 3 is used to fuse the high-level feature The up-sampling recovers the pixels, and then combines with the low-level feature f i r / d The fourth layer is spliced with itself, thereby splicing the adjacent f i r / d And Fusion into F i r / d Then, the fourth layer is spliced with itself, thereby splicing the adjacent f And F i i∈(1,4); for F i Calculate the spatial coordinate attention once, multiply the coordinate attention map obtained by F i to obtain the coordinate attention feature Then calculate the spatial attention once, multiply the spatial attention map obtained by F i to obtain the multi-level cross-modal fusion feature 5. The RGB-D saliency detection method based on progressive multi-scale feature fusion according to claim 4, characterized in that, Step 3 specifically includes: the cross attention fusion module includes the calculation of coordinate attention, the calculation of spatial attention, and the splicing and convolution operation; The cross-attention fusion module fuses the high-level features of the RGB stream and the depth stream of each stage Upsampling restores its pixels, and then combines with the low-level feature f i r / d The highest layer is spliced with itself, and the adjacent layer splicing feature F is obtained through convolution operation i r / d The preliminary stage splicing feature F is obtained through splicing operation i ; then, F i Calculate the coordinate attention, and the specific operation is one-dimensional horizontal pooling and one-dimensional vertical pooling on F i ; the pooled feature F i h / w , i∈(1,4) is spliced, and the global feature is extracted through convolution; then, the global feature is divided, the horizontal and vertical attention weight matrices are calculated, the two matrices are multiplied to obtain the coordinate attention map, and then multiplied with F i to obtain the coordinate attention feature Then, the spatial attention is calculated, and the specific operation is to perform global maximum pooling on the feature map, then extract the feature through Con7x7 convolution operation, and extract the spatial attention map through sigmoid activation function, and finally multiply with F i to obtain the multi-level cross-modal fusion feature The formula of this step is as follows: F i h / w = CDA h / w (ConvBS)p h (F i ),p w (F i ))) (3) where ConvBR(·) is a cascade operation of convolution, normalization and ReLu function, Up(·) is an up-sampling operation, Cat(·) is a concatenation operation, SA(·) is a spatial attention computation operation, CDA h / w (·) is a horizontal and vertical coordinate attention map generation operation, ConvBS(·) is a convolution, normalization and Sigmoid activation function cascade operation, is an element-wise multiplication operation. 6.The method of claim 1, wherein, In step 4, the spatial progressive fusion module starts from the first stage of the backbone network, and sequentially fuses the cross-modal fusion features of each stage After downsampling, the spatial attention is calculated once, and the spatial attention map is integrated into the adjacent high-level cross-modal fusion features In this way, the global features of different scales are progressively aggregated to obtain the final RGB-D image coding features 7.The method of claim 6, wherein, The specific method of step 4 is: the spatial progressive fusion module includes the calculation of spatial attention, down sampling and residual connection operation, and the specific implementation is as follows: Low-level cross-modal fusion features Calculate spatial attention once after downsampling; Wherein the spatial attention calculation carries out a global maximum pooling on the feature map, extracts features through convolution operation, and extracts a spatial attention map through a Sigmoid activation function; the spatial attention map extracted from is fused into in a residual connection manner, three spatial progressive fusion modules are connected in series, the aggregation of the four-stage cross-modal fusion features is realized, and the final RGB-D image encoding feature The formula of this step is as follows: where SA(·) is a spatial attention computation operation, Down(·) is a down-sampling operation, is an element-wise multiplication operation, and is an element-wise addition operation. 8.The RGB-D saliency detection method based on progressive multi-scale feature fusion of claim 1, characterized in that, In step 5, the encoded features obtained in step 3 are first upsampled and convolved to restore the pixel and channel to the same feature shape as the third layer features, obtaining In step 5, the encoded features obtained in step 3 are first upsampled and convolved to restore the pixel and channel to the same feature shape as the third layer features, obtaining In step 5, the encoded features obtained in step 3 are first upsampled and convolved to restore the pixel and channel to the same feature shape as the third layer features, obtaining The final result is obtained as the saliency prediction result.​​​​​​ 9.The method of claim 8, wherein, Step 5 specifically includes: the weight fusion module includes the calculation of channel attention weight and the calculation of splicing, convolution, normalization and activation function operation, and the specific implementation process is as follows: The encoded features obtained in step 3 Perform upsampling and a convolution to restore its pixel and channel numbers to the same The same characteristic shape is obtained Then use the weight fusion module to and Splicing; calculate its channel weight p through the channel attention module i ,i∈(1,3); perform a global average pooling on the input features, expand the dimension through the linear mapping layer, organize the data with the ReLu activation function, restore the dimension through the linear mapping, and extract the channel weights with the Sigmoid activation function; use the channel weights for weighted splicing and get The progressive convolution module comprises one depth separable convolution and two point convolution operations, and the specific implementation process is as follows: for the weight spliced feature map obtained by the weight fusion module, the channel number is compressed through convolution, adjusted through batch normalization and GeLu function, and then up-sampling is performed to restore the pixel size; the depth separable convolution is used to extract the information of each channel, and then the channel expansion and channel compression two point convolutions are used to extract the overall channel features, and then the saliency prediction result is obtained The three sets of paired weight fusion modules and progressive convolution modules are used to sequentially fuse the features of the three groups of feature maps to obtain the final feature map which is taken as the saliency prediction result The formula is as follows: wherein CA(·) is a calculation channel attention operation, Down(·) is a down-sampling operation, ConvBR(·) is a joint operation of convolution plus batch normalization plus ReLu activation function, BR(·) is a joint operation of batch normalization plus ReLu activation function, Sigmoid(·) is a Sigmoid activation function, p i is a channel weight, is a feature after fusing channel weights, is an element-wise addition operation, is a matrix multiplication operation; ConvBG(·) is a joint operation of convolution plus batch normalization plus GeLu activation function, DwConv(·) is a depth separable convolution operation, Pw1Conv(·) is an expansion point convolution operation, and Pw2Conv(·) is a compression point convolution operation.

10. A rail surface defect detection method using the saliency detection method based on progressive multi-scale feature fusion of RGB-D according to any one of claims 1-8, characterized in that: Use the industrial RGB-D data set NEU-RSDDSAUG as the task data set, construct and train the model; Use the trained model to detect defects on the collected rail surface image.

Images