RGB-D saliency detection method based on progressive multi-scale feature fusion

Abstract

The invention discloses an RGB-D saliency detection method based on progressive multi-scale feature fusion. The RGB-D saliency detection method comprises the following steps: 1) acquiring an RGB-D data set for training and testing a task; 2) selecting a double-flow Swin-Transform model as a trunk network, and extracting RGB features and depth features of four stages during model training; and 3) cross-modal fusion features of the four stages are obtained through a cross-attention fusion module. And 4) aggregating the cross-modal fusion features of the four stages into a final decoding feature by using a spatial progressive fusion module. And the cross-modal fusion features of the first three stages are sequentially fused into the decoding features through three weight fusion modules. And 5) after each weight fusion module, performing feature extraction on the feature map after weight fusion by using a progressive convolution module. And taking the output feature map of the last progressive convolution module as a result of RGB-D saliency target detection. The self-attention mechanism of the method can directly model the relationship between different positions on the feature map without being limited by the distance, so that the long-distance dependency relationship in the image is effectively captured.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and more specifically to an RGB-D saliency detection method based on progressive multi-scale feature fusion Background Art

[0002] RGB-D salient object detection (SOD) is a computer vision task that aims to achieve accurate segmentation of salient objects by analyzing the complementary information of RGB images and depth images. Currently, most RGB-D SOD models adopt a single-stream or two-stream structure in the backbone network design, among which the two-stream structure is widely adopted due to its advantage in capturing cross-modal information. Existing models usually extract image features based on convolutional neural networks (CNNs) and achieve good detection results. However, the feature extraction method of CNN based on sliding windows has inherent limitations, which makes it difficult to effectively capture global features in the image, resulting in insufficient modeling ability of long-range contextual dependencies, thus limiting the detection performance.

[0003] With the introduction of Vision Transformer (ViT), its powerful global modeling capability provides new ideas for salient object detection. Some researchers have tried to replace the traditional CNN backbone with a Transformer-based backbone to further improve the detection effect. However, how to effectively fuse the complementary information of RGB features and deep features is still one of the core problems in the field of RGB-DSOD. Most existing methods achieve cross-modal feature fusion through multi-path fusion, cross-level interaction, and adding additional feature enhancement modules, but these methods generally have problems such as high computational overhead, feature redundancy, and insufficient remote cross-modal feature interaction.

[0004] On the other hand, in order to adapt to the diversity of target sizes and environments in images, multi-scale feature fusion technology has become one of the key technologies to improve SOD performance. Existing studies usually enhance multi-scale features through attention mechanisms, spatial pyramid pooling (ASPP) or context interaction modules. However, in the multi-scale feature fusion stage, direct feature concatenation or multiple sampling operations may lead to feature redundancy, noise introduction and information loss, which in turn affects the prediction accuracy of salient targets.

[0005] In summary, the existing RGB-D SOD methods have limitations in cross-modal fusion and multi-scale feature fusion. There is an urgent need for a new method that can efficiently extract and fuse multi-modal and multi-scale information to improve the accuracy and robustness of salient object detection. Summary of the invention

[0006] In view of the above-mentioned practical problems and in order to solve the shortcomings of the existing technology, this paper carefully analyzes the problem technology field to which this type of problem belongs, and proposes an RGB-D saliency detection method based on progressive multi-scale feature fusion.

[0007] The technical solution of the present invention is as follows:

[0008] The RGB-D saliency detection method based on progressive multi-scale feature fusion includes the following steps:

[0009] Step 1: Get the training set and test set required for the task.

[0010] Step 2: Use the two-stream Swin-transformer network as the backbone network to extract RGB image features and depth image features of different scales at four stages during the model training process, denoted as f i r and f i d ,i∈(1,4).

[0011] Step 3: Use the Cross Attention Fusion Module (CAFM) to combine the output features f of adjacent layers in the RGB branch and the depth branch respectively. i r / d and Fusion into F i r / d , i∈(1,4). Then and Perform cross-modal feature fusion to obtain multi-level cross-modal fusion features

[0012] Step 4: Use the Spatial Progressive Fusion Module (SPFM) to combine the cross-modal fusion features obtained in step 3 The low-level spatial information is extracted progressively from low-level features to high-level features and integrated into high-level features to obtain the final RGB-D image encoding features.

[0013] Step 5: Upsample and compress the channel to obtain the encoded features The cross-modal features obtained in step 3 The feature dimensions of Starting from the third stage, the weight fusion module (WFM) is used to and Perform weighted splicing on the channel and use the difference convolution module (PCM, Progressive Convolution Module) to decode the splicing results. The final As a result of significant prediction.

[0014] Furthermore, 1485, 700, and 800 groups of images were selected from the public datasets NJU2K, NLPR, and DUT, respectively, for a total of 2985 groups of images as training sets, and each group of images includes three images: RGB images, depth images, and saliency true images. 135, 400, 100, 500, 300, and 929 groups of images were selected from the public datasets DES, DUT, LFSD, NJU2K, NLPR, and SIP, respectively, as comparative test sets. During implementation, the training and test datasets can be replaced for specific application objects.

[0015] Furthermore, in step 2, we use the symmetric two-stream Swin-Transformer of the Swin-B version as the backbone network to process RGB images and depth images respectively. We extract RGB image features and depth image features of different scales at four stages during the model training process, which are denoted as f i r and f i d .

[0016] Furthermore, in step 3, the cross attention fusion module is used to combine high-level features Upsample to restore its pixels and then combine them with the low-level features f i r / d Splice (the fourth layer is spliced ​​with itself), so that the adjacent f i r / d and Fusion into F i r / d Next, for F i r / d Calculate the spatial coordinate attention once, and you will get the coordinate attention map and F i r / d Multiply them together to get the coordinate attention feature map. Then calculate the spatial attention again, and get the spatial attention map and F i r / d Multiply to get multi-level cross-modal fusion features

[0017] Furthermore, in step 4, the spatial progressive fusion module starts from the first stage of the backbone network and sequentially integrates the cross-modal fusion features of each stage. After downsampling, the spatial attention is calculated once, and its spatial attention map is integrated into the adjacent high-level cross-modal fusion features. In this way, the global features of different scales are gradually aggregated to obtain the final RGB-D image encoding features.

[0018] Furthermore, in step 5, the coding features obtained in step 3 are first Upsampling and convolution will The pixels and channels are restored to the same characteristic shape as the third layer features, and we get The weight fusion module is then used to combine the cross-modal fusion features of the third stage obtained in step 3 After the channel weights are calculated by channel attention, the weighted splicing is performed again. The weighted splicing result is then fused with features through the residual convolution module to obtain Using three sets of paired weight fusion modules and progressive convolution modules, Integration get Will eventually get This is taken as the result of the significant prediction.

[0019] The second aspect of the present invention relates to a rail surface defect detection method that applies the RGB-D saliency detection method based on progressive multi-scale feature fusion of the present invention, using the industrial RGB-D dataset NEU-RSDDSAUG as a task dataset to construct and train a model; and using the trained model to perform defect detection on the collected rail surface images.

[0020] The beneficial effects of the present invention are as follows:

[0021] 1. The present invention uses the Swin-Transformer network as the backbone network. Its built-in self-attention mechanism can directly model the relationship between different positions on the feature map without being restricted by distance, thereby effectively capturing long-distance dependencies in the image.

[0022] 2. The design of CAFM adopts the tandem use of coordinate attention and spatial attention, which effectively focuses on and mines the long-distance dependency between the target and the background in the cross-modal features, and obtains a wider range of attention information without adding additional feature enhancement modules, thereby achieving full cross-modal integration of RGB features and depth features.

[0023] 3. When designing modules such as CAFM, SPFM, WFM, and RCM, different multi-scale fusion strategies are used before cross-modal fusion, after cross-modal fusion, and in the decoding stage to condense and extract features in space and channels. This method enables the model to fully integrate global context information and improve the detection accuracy of objects of different scales. At the same time, in the encoding and decoding stages, the progressive weighting strategy also effectively alleviates the problems of feature loss and redundancy in the feature sampling and splicing process.

[0024] Figure 1 It is a process framework diagram of the present invention.

[0025] Figure 2 It is a schematic diagram of the network structure of the present invention.

[0026] Figure 3 It is a schematic diagram of the structure of the channel cross attention fusion module (CAFM).

[0027] Figure 4 It is a schematic diagram of the spatial progressive fusion module (SPFM) structure.

[0028] Figure 5 It is a schematic diagram of the structure of the weight fusion module (WFM).

[0029] Figure 6 It is a schematic diagram of the progressive convolution module (PCM) structure. DETAILED DESCRIPTION

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] This embodiment provides a rail surface defect detection method based on an RGB-D saliency detection method with progressive multi-scale feature fusion.

[0032] Rails are key infrastructure for railway transportation. The surface defects of rails (such as holes, grooves, scars, protrusions, etc.) are directly related to the safety and efficiency of train operation. Therefore, rail defect detection has become a vital task in railway maintenance. Traditional rail detection methods mainly rely on manual inspections or computer vision algorithms based on a single modality. Although they have achieved certain results, they still have problems of insufficient accuracy and poor robustness when facing complex background interference, locating small defects, and detecting multiple types of defects at the same time.

[0033] In order to adapt to the diversity of rail defects in size, shape and distribution, multi-scale feature fusion technology has become one of the key technologies to improve detection performance. Existing studies usually enhance multi-scale features through attention mechanisms, spatial pyramid pooling (ASPP) or context interaction modules. However, in the multi-scale feature fusion stage, direct feature concatenation or multiple sampling operations may lead to feature redundancy, noise introduction and information loss, which in turn affects the detection accuracy and positioning ability of the defect area.

[0034] In summary, the existing RGB-D SOD method has certain limitations in cross-modal fusion and multi-scale feature fusion. In view of the specific needs of rail defect detection, a new method that can efficiently extract and fuse multi-modal and multi-scale information is urgently needed to improve the accuracy and robustness of defect detection and provide strong support for the safety of railway transportation.

[0035] like Figure 1 As shown, the method of this embodiment includes the following specific steps:

[0036] Step 1. Use the industrial RGB-D dataset NEU-RSDDSAUG as the task dataset. Use MATLAB software to segment the image through random grids to complete data enhancement. After filtering the images with poor quality, 1862 pairs of images were sorted out. Use the LabelImg annotation tool to annotate the defects of four types: holes, grooves, scars, and protrusions. Divide 1490 pairs of images into training sets and 372 pairs of images into test sets in a ratio of 8:2.

[0037] Step 2: Figure 2 As shown in Figure 1, the symmetrical two-stream Swin-Transformer of the Swin-B version is used as the backbone network to process RGB images and depth images respectively. The RGB image features and depth image features of different scales at four stages in the model training process are extracted and recorded as f i r and f i d ,i∈(1,4).

[0038] Step 3: Figure 3 As shown in the figure, the cross attention fusion module mainly includes the calculation of coordinate attention, the calculation of spatial attention, and several splicing and convolution operations. The function of the cross attention fusion module is to combine the high-level features of the RGB stream and the depth stream of each stage. Upsample to restore its pixels and then combine them with the low-level features f i r / d Perform splicing (splice the highest layer with itself), and obtain the adjacent layer splicing feature F through convolution operation i r / d,i∈(1,4), the preliminary stage splicing feature F is obtained through the splicing operation i ,i∈(1,4). Next, for F i Calculate the coordinate attention, the specific operation is to F i r / d Perform one-dimensional horizontal pooling and one-dimensional vertical pooling. i h / w ,i∈(1,4) for splicing, and extract global features through convolution. Then, the global features are segmented to calculate the horizontal and vertical attention weight matrices, and the two matrices are multiplied to obtain the coordinate attention map, which is then combined with F i Multiply to get the coordinate attention feature Next, we calculate the spatial attention. The specific operation is to perform a global maximum pooling on the feature map, extract features through the Con7x7 convolution operation, extract the spatial attention map through the sigmoid activation function, and finally add it to F i r / d Multiply to get multi-level cross-modal fusion features

[0039] The formula for this step is as follows:

[0040]

[0041] ConvBR(·) is the concatenation of convolution, normalization and ReLu functions, Up(·) is the upsampling operation, Cat(·) is the concatenation operation, SA(·) is the spatial attention calculation operation, and CDA h / w (·) horizontal and vertical coordinate attention map generation operation, ConvBS (·) is the cascade operation of convolution, normalization and Sigmoid activation function, It is an element-by-element multiplication operation.

[0042] Step 4: If Figure 4 As shown in Figure 1, the spatial progressive fusion module mainly includes operations such as spatial attention calculation, downsampling and residual connection. Its specific implementation is: the cross-modal fusion features of the low layer After downsampling, the spatial attention is calculated once. The specific operation of the spatial attention calculation is to perform a global maximum pooling on the feature map, extract features through convolution operations, and extract the spatial attention map through the Sigmoid activation function. The spatial attention map extracted from In the example, three spatial progressive fusion modules are connected in series to achieve the aggregation of four-stage cross-modal fusion features and obtain the final RGB-D image encoding features.

[0043] The formula for this step is as follows:

[0044]

[0045] Where SA(·) is the spatial attention calculation operation, Down(·) is the downsampling operation, is an element-by-element multiplication operation, It is an element-by-element addition operation.

[0046] Step 5: Figure 5 As shown in Figure 1, the weight fusion module mainly includes the calculation of channel attention weights and several operations such as splicing, convolution, normalization and activation. The specific implementation process is as follows: Upsampling and convolution are performed to restore the number of pixels and channels to the same The same characteristic shape is obtained Then use the weight fusion module to and Splicing. Through the channel attention module, calculate its channel weight p i ,i∈(1,3). The specific operation of channel weight calculation is: perform a global average pooling on the input features, expand the dimension through the linear mapping layer, organize the data with the relu activation function, restore the dimension through linear mapping, and extract the channel weight with the sigmoid activation function. After obtaining the channel weight, use the channel weight to weight the splicing and get like Figure 6 As shown in the figure, the progressive convolution module mainly includes a depth-separable convolution and two point convolution operations. The specific implementation process is as follows: for the weighted splicing feature map obtained by the weight fusion module, the number of channels is compressed through convolution, adjusted through batch normalization and GeLu function, and then upsampled to restore the pixel size. The information of each channel in the , and then use channel expansion and channel compression two point convolution to extract the overall channel features, and then get Using three sets of paired weight fusion modules and progressive convolution modules, Integration get Will eventually get It is used as the result map of rail defect detection. In the rail defect detection task, the trained model is used to process and identify the real-time collected rail surface images. According to the detection result map of the model, it is judged whether there are defects on the rail surface and the shape and location of the defects.

[0047] The formula for this step is as follows:

[0048]

[0049] Where CA(·) is the channel attention calculation operation, Down(·) is the downsampling operation, ConvBR(·) is the joint operation of convolution plus batch normalization plus ReLu activation function, BR(·) is the joint operation of batch normalization plus ReLu activation function, Sigmoid(·) is the Sigmoid activation function, p i is the channel weight, is the feature after fusion of channel weights, is an element-by-element addition operation, and × is a matrix multiplication operation. ConvBG(·) is a joint operation of product plus batch normalization plus GeLu activation function, DwConv(·) is a depth-wise separable convolution operation, Pw1Conv(·) is an extended point convolution operation, and Pw2Conv(·) is a compressed point convolution operation.

[0050] Step 6: Use a depth camera to inspect the rails, collect RGB and depth image information of the rail surface in real time, use the trained model to process and identify the collected images, and determine whether there are defects in the identification results.

[0051] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. An RGB-D saliency detection method based on progressive multi-scale feature fusion, characterized by: The following steps are involved: Step 1: Obtain the training set and test set required for the task; Step 2: Use the two-stream Swin-transformer network as the backbone network to extract RGB image features and depth image features of different scales at four stages during model training, which are recorded as and i∈(1,4); Step 3: Use the Cross Attention Fusion Module (CAFM) to combine the output features of adjacent layers in the RGB branch and the depth branch and Fusion i∈(1,4). Then and Perform cross-modal feature fusion to obtain multi-level cross-modal fusion features i∈(1,4); Step 4: Use the spatial progressive fusion module (SPFM) to fuse the cross-modal features obtained in step 3 The low-level spatial information is extracted progressively from low-level features to high-level features and integrated into high-level features to obtain the final RGB-D image encoding features. ( i∈(0,4)); Step 5: Upsample and compress the channel to obtain the encoded features The cross-modal features obtained in step 3 The feature dimensions of Starting from the third stage, the weight fusion module (WFM) is used to and Perform weighted splicing on the channel and use the difference convolution module (PCM) to decode the splicing result to obtain The final As a result of significant prediction.

2. The RGB-D saliency detection method based on progressive multi-scale feature fusion according to claim 1, characterized in that: The specific method of step 1 is: select 1485, 700, and 800 groups of images from the public datasets NJU2K, NLPR, and DUT, respectively, for a total of 2985 groups of images as training sets, and each group of images includes three images: RGB images, depth images, and saliency true images. Select 135, 400, 100, 500, 300, and 929 groups of images from the public datasets DES, DUT, LFSD, NJU2K, NLPR, and SIP, respectively, as comparative test sets; in implementation, the training and test datasets can be replaced for specific application objects.

3. The RGB-D saliency detection method based on progressive multi-scale feature fusion according to claim 1, characterized in that: The specific method of step 2 is: use the symmetrical two-stream Swin-Transformer of Swin-B version as the backbone network to process RGB images and depth images respectively. Extract RGB image features and depth image features of different scales at four stages during model training, which are recorded as and i∈(1,4).

4. The RGB-D saliency detection method based on progressive multi-scale feature fusion according to claim 1, characterized in that: In step 3, the role of the cross attention fusion module is to combine high-level features Upsampling restores its pixels and then combines them with the low-level features Splice (the fourth layer is spliced ​​with itself), so that the adjacent and Fusion Next, splicing and Get F i i∈(1,4); for F i Calculate the spatial coordinate attention once, and you will get the coordinate attention map and F i Multiply to get the coordinate attention feature i∈(1,4); calculate the spatial attention again, and get the spatial attention map and F i Multiply to get multi-level cross-modal fusion features i∈(1,4).

5. The RGB-D saliency detection method based on progressive multi-scale feature fusion according to claim 4, characterized in that: Step 3 specifically includes: the cross attention fusion module includes the calculation of coordinate attention, the calculation of spatial attention, and the concatenation and convolution operations; The cross attention fusion module combines the high-level features of the RGB stream and the depth stream at each stage Upsampling restores its pixels and then combines them with the low-level features Perform splicing (splice the highest layer with itself), and obtain the adjacent layer splicing features through convolution operation The preliminary stage splicing feature F is obtained through the splicing operation i ; Then, for F i Calculate the coordinate attention, the specific operation is to F i Perform one-dimensional horizontal pooling and one-dimensional vertical pooling. i∈(1,4) for splicing, and extract global features through convolution; then, segment the global features, calculate the horizontal and vertical attention weight matrices, multiply the two matrices to obtain the coordinate attention map, and then add it to F i Multiply to get the coordinate attention feature Next, we calculate the spatial attention. The specific operation is to perform a global maximum pooling on the feature map, extract features through the Con7x7 convolution operation, extract the spatial attention map through the sigmoid activation function, and finally add it to F i Multiply to get multi-level cross-modal fusion features The formula for this step is as follows: ConvBR(·) is the concatenation of convolution, normalization and ReLu functions, Up(·) is the upsampling operation, Cat(·) is the concatenation operation, SA(·) is the spatial attention calculation operation, and CDA h / w (·) horizontal and vertical coordinate attention map generation operation, ConvBS (·) is the cascade operation of convolution, normalization and Sigmoid activation function, It is an element-by-element multiplication operation.

6. The RGB-D saliency detection method based on progressive multi-scale feature fusion according to claim 1, characterized in that: In step 4, the spatial progressive fusion module starts from the first stage of the backbone network and sequentially integrates the cross-modal fusion features of each stage. After downsampling, the spatial attention is calculated once, and its spatial attention map is integrated into the adjacent high-level cross-modal fusion features. In this way, the global features of different scales are gradually aggregated to obtain the final RGB-D image encoding features.

7. The RGB-D saliency detection method based on progressive multi-scale feature fusion according to claim 6, characterized in that: The specific method of step 4 is: the spatial progressive fusion module includes the calculation of spatial attention, downsampling and residual connection operations, which are specifically implemented as follows: Fusion of low-level cross-modal features Calculate spatial attention once after downsampling; The spatial attention calculation performs a global maximum pooling on the feature map, extracts features through convolution operations, and extracts the spatial attention map through the Sigmoid activation function; The spatial attention map extracted from In the above, three spatial progressive fusion modules are connected in series to achieve the aggregation of cross-modal fusion features in four stages and obtain the final RGB-D image encoding features. The formula for this step is as follows: Where SA(·) is the spatial attention calculation operation, Down(·) is the downsampling operation, is an element-by-element multiplication operation, and ⊕ is an element-by-element addition operation.

8. The RGB-D saliency detection method based on progressive multi-scale feature fusion according to claim 1, characterized in that: In step 5, the encoding features obtained in step 3 are first Upsampling and convolution will The pixels and channels are restored to the same characteristic shape as the third layer features, and we get The weight fusion module is then used to combine the cross-modal fusion features of the third stage obtained in step 3 After the channel weights are calculated by channel attention, weighted splicing is performed again; the weighted splicing result is then fused through the residual convolution module to obtain Using three sets of paired weight fusion modules and progressive convolution modules, Integration get Will eventually get This is taken as the result of the significant prediction.

9. The RGB-D saliency detection method based on progressive multi-scale feature fusion according to claim 8, characterized in that: Step 5 specifically includes: the weight fusion module includes the calculation of channel attention weights and the calculation operations of splicing, convolution, normalization and activation functions. The specific implementation process is as follows: The encoded features obtained in step 3 Upsampling and convolution are performed to restore the number of pixels and channels to the same The same characteristic shape is obtained Then use the weight fusion module to and Splicing; through the channel attention module, calculate its channel weight p i ,i∈(1,3); perform a global average pooling on the input features, expand the dimension through the linear mapping layer, organize the data with the ReLu activation function, restore the dimension through linear mapping, and extract the channel weights with the Sigmoid activation function; use the channel weights for weighted concatenation and get i∈(1,3); The progressive convolution module contains a depth-separable convolution and two point convolution operations. Its specific implementation process is as follows: for the weighted splicing feature map obtained by the weight fusion module, the number of channels is compressed through convolution, adjusted through batch normalization and GeLu function, and then upsampled to restore the pixel size; the depth-separable convolution is used to extract The information of each channel in the , and then use channel expansion and channel compression two point convolution to extract the overall channel features, and then get Using three sets of paired weight fusion modules and progressive convolution modules, Integration get Will eventually get Take it as the result of significant prediction; The formula is as follows: Where CA(·) is the channel attention calculation operation, Down(·) is the downsampling operation, ConvBR(·) is the joint operation of convolution plus batch normalization plus ReLu activation function, BR(·) is the joint operation of batch normalization plus ReLu activation function, Sigmoid(·) is the Sigmoid activation function, p i is the channel weight, is the feature after fusion of channel weights, ⊕ is the element-by-element addition operation, and × is the matrix multiplication operation. ConvBG(·) is the joint operation of convolution plus batch normalization plus GeLu activation function, DwConv(·) is the depth-wise separable convolution operation, Pw1Conv(·) is the extended point convolution operation, and Pw2Conv(·) is the compressed point convolution operation.

10. A rail surface defect detection method using the RGB-D saliency detection method based on progressive multi-scale feature fusion as claimed in claim 1, characterized in that: Use the industrial RGB-D dataset NEU-RSDDSAUG as the task dataset to construct and train the model; Use the trained model to perform defect detection on the collected rail surface images.

Images