A method for recognizing a salient target based on multi-modal feature fusion

Abstract

The application discloses a kind of based on multimodal fusion's salient target detection model, including three kinds of modal characteristics are extracted by vgg16 network.For being able to fully use the characteristics of different modal, the feature extracted in each layer is sent into feature fusion module, feature fusion module is divided into two parts, respectively, feature connection and feature fusion.Adjacent two layer features are used to generate enhanced features, and the spatial attention module generated by them is used to enhance the decoder output features of the previous layer, so that more salient feature map is obtained.The output of the fourth layer S module is connected back to the input of each decoder D module and added to other inputs.This can fully utilize high-level semantic features multiple times, making the fusion more complete.In addition, the application is provided with a loss supervision at the output end of each encoder to train each module more finely, so as to obtain a very accurate salient target detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a salient object recognition method based on multi-modal feature fusion. BACKGROUND

[0002] Salient object detection is an important research direction in the field of computer vision, aiming to accurately detect and locate salient objects that are obviously different from the background from images or videos. These salient objects are usually the focus of user attention, and therefore have important value in many applications, such as image retrieval, autonomous driving, video monitoring, medical image analysis, etc. Significant progress has been made in salient object recognition for general rgb images or salient object recognition for dual modal images (rgb&D or rgd&T).

[0003] For single-modal salient object recognition, researchers have been exploring various effective feature extraction methods, including low-level features based on traditional image processing techniques (such as color, texture, and edge, etc.), and high-level features based on deep learning (such as convolutional layer features in convolutional neural networks). In recent years, deep learning methods have made significant progress in salient object detection. Semantic segmentation has made great progress in the past decade and is widely used in autonomous driving and medical analysis. However, due to the physical limitations of image sensors, most single-modal semantic segmentation models are limited in some application scenarios, such as low-light environments, fog, and smoke. Therefore, multi-modal semantic segmentation has received extensive attention in recent years.

[0004] In the field of daily salient object recognition, in addition to RGB images as the main recognition processing object, thermal infrared (TIR) images are also often used as the processing object. Because thermal infrared images reflect the surface temperature information of objects, they can highlight vehicles and pedestrians with higher temperatures than the surrounding environment. Some people introduce TIR images as a supplement to improve the model's ability to handle challenging urban driving scenarios, which has spurred the rise of RGB-hertir (RGB-T) semantic segmentation.

[0005] Currently, most salient object detection methods on the market are dual-modal, and triple-modal salient object work is still in its early stages. The existing rgb, thermal, depth triple-modal fusion salient object recognition methods mostly have poor performance, the edge detection is not clear enough, the recognition is not accurate enough, and the detection effect cannot meet our needs. Therefore, there are still many methods to be explored for better fusion of multi-modal features. SUMMARY

[0006] The present application proposes a salient object recognition method based on multi-modal feature fusion to optimize edge detection and achieve better salient object detection effect, thereby meeting market demand.

[0007] To solve the above technical problems, the technical scheme of the present application is:

[0008] A significant target recognition method based on multi-modal feature fusion, comprising the following steps:

[0009] S1, constructing an encoder and extracting the features of rgb, thermal and depth images using the constructed encoder;

[0010] S2, constructing a feature fusion module, and obtaining i-layer primary fusion features through multi-modal information fusion by the feature fusion module,

[0011] The feature fusion module includes a feature connection network and a feature cross network, the feature connection network is composed of three parallel SE+CBR modules, the SE module is composed of a global average pooling layer, a convolution layer, a Relu activation function layer and a sigmoid function layer; the CBR module is composed of a convolution layer, a regularization layer and a Relu layer; the feature cross module is composed of a CBR module and a plurality of layers of dilated convolution with different expansion rates;

[0012] S3, constructing a fusion module S, and mutually fusing the primary fusion features of adjacent feature layers by the fusion module S;

[0013] The fusion module S is composed of two spatial attentions, the first spatial attention is an average pooling in the channel dimension, specifically, the average value of each channel of the feature map is obtained to obtain a spatial attention map with a channel number of 1;

[0014] S4, decoding the secondary fusion features layer by layer after fusion, and the specific decoding method is as follows:

[0015] Firstly, unlike other layers, the fifth layer primary fusion feature F5 obtained in step S2 is directly sent into the decoding block D, and after Dropout, CBR and Upsample processing, it is marked as D5, which is represented as:

[0016] D5=Upsample(CBR(CBR(Dropout(F5)))

[0017] For the fourth decoding block, the input thereof is the output of the fourth fusion module S, which can be represented as:

[0018] D4=Upsample(CBR(CBR(Dropout(S4)))

[0019] For other layers, it can be represented as:

[0020] D i=Upsample(CBR(CBR(Dropout(S i +S4)))) (i=1,2,3)

[0021] Among them S i is the output of the i-th layer S module, and the output of the last layer decoding block is the saliency map.

[0022] Preferably, the encoder is composed of three VGG-16 networks that do not share parameters.

[0023] Preferably, there are 4 dilated convolutions in the feature cross network, and their dilation rates are 1, 3, 5, and 7 respectively.

[0024] Preferably, the multimodal information fusion method in step S2 is expressed as follows:

[0025]

[0026]

[0027]

[0028] f ci =CAT((F Vi , F Ti , F Di ))

[0029] F i =Conv 3*3 (CAT(φ1(f ci ),φ3(f ci ),φ5(f ci ),φ7(f ci ))) (i=1,2,3,4,5)

[0030] Among them, CBR represents convolution layer, regularization layer, Relu layer, X Vi 、X Ti 、X Di Represents the features of V, T, and D extracted from the i-th layer respectively Represents the global average pooling function, CAT represents the connection operation, Conv a*a Represents a convolution operation with a convolution kernel side length of a, φ a represents a convolution with a dilation rate of a.

[0031] Preferably, the method of step S3 is:

[0032] Five layers of primary fusion features are obtained through the feature fusion module, and the primary fusion features of adjacent layers and the output features of the decoding block D of the previous layer are sent to the fusion module S;

[0033] First, the primary fusion features of the two adjacent layers are added, then the features are optimized by the channel average spatial attention, the output of the previous layer is added to the current layer, and finally another spatial attention is used to further optimize the features, and the process can be represented as:

[0034] S' i = Conv 3*3 (Upsample(F i+1 ))+F i

[0035] S i = SA(β(S' i )×(S' i +D i+1 )) (i = 1, 2, 3, 4)

[0036] Where Upsanple(·) is bilinear upsample two times, F i is the output of the i-th layer multi-modal information fusion, and D i+1 is the output of the previous layer decoding module.

[0037] As preferred, the output of the fourth fusion module S is backwardly connected to the input end of each decoding block D and added to the output of the fusion module S, and finally sent into the decoding block D. In this way, the high-level semantic features can be fully utilized multiple times, so that the fusion is more sufficient. In addition, the present application is provided with a loss supervision at the output end of each encoder to train each module in a more refined manner, so as to obtain a very accurate salient object detection result.

[0038] The present application has the following characteristics and beneficial effects:

[0039] The three modal features of the present application are extracted by the vgg16 network. The adjacent fusion features with similarity are significantly enhanced by the S module, so that the salient object is more prominent. In the decoding stage, multi-level information is fused, and the quality of the fused features is higher. Through the multi-layer jump connection, the fusion is more sufficient, and the model performance is higher than that of the previous three modal fusion models. The salient object detection effect is good. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1The framework diagram of the method of the present application.

[0042] Figure 2 The three-modal feature fusion module F of the method of the present application.

[0043] Figure 3 The adjacent feature fusion module S of the method of the present application.

[0044] Figure 4 The decoding block D of the method of the present application.

[0045] Figure 5 The effect diagram of the method of the present application (the first column is the RGB image, the second column is the label image, and the third column is our predicted image). DETAILED DESCRIPTION

[0046] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0047] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0048] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood through specific circumstances.

[0049] The present application provides a significant target recognition method based on multi-modal feature fusion, which includes an encoder composed of three VGG-16 neural networks, an innovative three-modal fusion method, a mutual enhancement of adjacent layer feature fusion, and a multi-level jump guidance.

[0050] The specific implementation of the present application is divided into three stages, namely the preparation and preprocessing of the data set, the prediction of the significant target, and the comparison of the results.

[0051] First, the three-modal data set we used is the VDT-2048 data set, which contains 1048 training pictures and 1000 test pictures. It also contains some challenging tests for each modality. It is currently the largest publicly available rgb, depth, thermal three-modal data set. The height and width of the original picture are 480 and 640, and after data preprocessing, it is cropped to 352x352 size, and the final prediction map is also restored to 480x640 size.

[0052] The significant target prediction process is described in detail as follows:

[0053] The steps are as follows, as shown in Figure 1

[0054] Step (1) extraction of multi-modal features, the specific method is as follows:

[0055] First, the encoder structure is constructed, and our encoder is constructed by three vgg16s that do not share parameters, which are used to extract the features of rgb, thermal and depth images respectively.

[0056] Step (2) multi-modal information fusion, the specific method is as follows:

[0057] In order to make full use of the features of different modalities, we propose a new simple and efficient feature fusion method. The features extracted from each layer are sent to the feature fusion module, which is divided into two parts, feature connection and feature cross. The feature connection part is composed of three parallel SE and CBR modules, where the SE module is a residual connection type composed of a global average pooling layer, a convolution layer, a Relu activation function layer, and a sigmoid function layer. CBR is composed of a convolution layer, a regularization layer, and a Relu layer. After the SE+CBR module, the features of the three modalities are connected. The feature cross module is composed of CBR and dilated convolution with different dilation rates, and the dilation rates of the dilated convolution are (1, 3, 5, 7) respectively. After feature connection and feature cross, the fused features of the three modalities are obtained, which are denoted as F1, F2, F3, F4, F5 respectively. The process can be represented as:

[0058]

[0059]

[0060]

[0061] fc​i = CAT((F Vi , F Ti , F Di ))

[0062] F i = Conv 3*3 (CAT(φ1(f ci ), φ3(f ci ), φ5(f ci ), φ7(f ci ))) (i = 1, 2, 3, 4, 5)

[0063] where CBR represents convolutional layer, regularization layer, Relu layer, X Vi , X Ti , X Di represent the features of V, T, D extracted by the i-th layer respectively represents global average pooling function. CAT represents concatenation operation. Conv a*a represents convolution operation with kernel size a. φ a represents convolution with dilation rate a. In this project, a is equal to 1, 3, 5, 7 respectively.

[0064] Step (3) fusion of adjacent feature layers, the specific method is as follows:

[0065] Five layers of features have been obtained through the fusion module. The features of adjacent layers and the decoding features of the previous layer are sent into the fusion module S. The fusion module S is composed of channel dimension average pooling spatial attention and traditional channel dimension maximum pooling spatial attention. First, the features of adjacent two layers are added, then the channel average pooling spatial attention is used to optimize the features, then the output of the previous layer is added to the current layer, and finally the traditional spatial attention is used to further optimize the features. The process can be represented as:

[0066] S′ i = Conv 3*3 (Upsample(F i+1 ))+F i

[0067] S i = SA(β(S′ i )×(S′ i +D i+1 )) (i = 1, 2, 3, 4)

[0068] where Upsanple(·) is bilinear up-sampling twice, F i is the output of the i-th layer multi-modal information fusion, and D i+1 is the output of the previous layer decoding module.

[0069] Step (4) decodes the fused features layer by layer. The decoding method is as follows:

[0070] Firstly, unlike other layers, we directly send the fifth layer feature F5 into the decoding block D, and after the Dropout, CBR, and Upsample processing, it is marked as D5. It is expressed as:

[0071] D5 = Upsample(CBR(CBR(Dropout(F5)))

[0072] For the decoding block of the fourth layer, the input is the output of the fourth layer fusion module S, which can be expressed as:

[0073] D4 = Upsample(CBR(CBR(Dropout(S4)))

[0074] For other layers, it can be expressed as:

[0075] D i = Upsample(CBR(CBR(Dropout(S i +S4)))) (i = 1, 2, 3)

[0076] Where S i is the output of the i-th fusion module S. The output of the last decoding block is the saliency map we get.

[0077] Specifically, as can be seen from the formula, the input of the first spatial attention in the first fusion module S is the primary fusion feature F1 and F2 to obtain S1', the input of the first spatial attention in the second fusion module S is the primary fusion feature F2 and F3 to obtain S2', the input of the first spatial attention in the third fusion module S is the primary fusion feature F3 and F4 to obtain S3', and the input of the first spatial attention in the fourth fusion module S is the primary fusion feature F4 and F5 to obtain S4'.

[0078] S1 is obtained by inputting S1' and D2 into the second spatial attention, S2 is obtained by inputting S2' and D3 into the second spatial attention, S3 is obtained by inputting S3' and D4 into the second spatial attention, and S4 is obtained by inputting S4' and D5 into the second spatial attention,

[0079] D5 is obtained by decoding F5, D4 is obtained by decoding S4, D3 is obtained by decoding S3 and S4, D2 is obtained by decoding S2 and S4, and D1 is obtained by decoding S1 and S4.

[0080] It can be seen that step S3 and step S4 are fused and decoded with each other, and then the final result, i.e., D1, is output.

[0081] As shown in the table, the first column is the RGB original image, the second column is the label of the salient object, and the third column is the final saliency map obtained by us. It can be seen that the saliency map predicted by us is very close to the label. Figure Five

[0082] Further, the present application innovatively uses the similarity between adjacent layers to enhance the features, and uses a simple and efficient three-modal feature fusion method, which fully utilizes the complementarity between multi-modal features. The fusion module S is designed independently to enhance the features of adjacent layers, and the fusion module S is embedded into the decoder while fusing the output of the previous layer decoding. The spatial attention mode of channel average is used in the fusion module S to constrain and optimize the adjacent layer features and the output of the previous layer decoding. The convolution kernel size of the convolution layer is 3x3, the step is 1, the upsample uses bilinear upsample, and the multiple is 2.

[0083] The output of the fourth fusion module S is backwardly jump-connected to the input end of each decoding block D and added to the output of the fusion module S, and finally sent into the decoding block D together.

[0084] The embodiments of the present application are described in detail above in combination with the drawings, but the present application is not limited to the described embodiments. For those skilled in the art, various changes, modifications, replacements and variations of the embodiments including components are made without departing from the principles and spirits of the present application, and still fall within the protection scope of the present application.​

Claims

1. A salient object recognition method based on multimodal feature fusion, characterized in that: The steps include: S1. Build an encoder and use the built encoder to extract features of RGB, thermal, and depth images; S2. Construct a feature fusion module, and perform multimodal information fusion through the feature fusion module to obtain i-layer primary fusion features. The feature fusion module includes a feature connection network and a feature cross network. The feature connection network consists of three parallel SE+CBR modules. The SE module consists of a global average pooling layer, a convolution layer, a Relu activation function layer, and a sigmoid function layer. The CBR module consists of a convolution layer, a regularization layer, and a Relu layer. The feature cross network consists of a CBR module and several layers of dilated convolutions with different dilation rates. S3, constructing four fusion modules S, and fusing the primary fusion features of adjacent feature layers through the four fusion modules S to obtain secondary fusion features; The fusion module S consists of two spatial attentions, the first of which is the average pooling of the channel dimension; S4. Decode the secondary fusion features layer by layer through a decoder. The decoder consists of five decoding blocks D. The specific decoding method is as follows: First, unlike other layers, the fifth layer primary fusion features obtained in step S2 are It is directly sent to the fifth decoding block D and marked as . Expressed as: )； For the fourth decoding block, its input is the output of the fourth fusion module S, which can be expressed as: )； For other layers, it can be expressed as: )，i=1,2,3； in is the output of the i-th fusion module S, and the output of the last decoding block is the obtained saliency map.

2. The method for identifying salient objects based on multimodal feature fusion according to claim 1, characterized in that: The encoder consists of three VGG-16 networks that do not share parameters.

3. The method for identifying salient objects based on multimodal feature fusion according to claim 1, characterized in that: There are 4 dilated convolutions in the feature cross network, and their dilation rates are 1, 3, 5, and 7 respectively.

4. The method for identifying salient objects based on multimodal feature fusion according to claim 1, wherein: The multimodal information fusion method in step S2 is expressed as follows: ))))； ))))； ))))； ； ，i=1,2,3,4,5； Among them, CBR represents convolution layer, regularization layer, and Relu layer. 、 、 Represents the features of V, T, and D extracted from the i-th layer respectively represents the global average pooling function, CAT represents the connection operation, Represents a convolution operation with a convolution kernel side length of a. represents a convolution with a dilation rate of a.

5. The method for identifying salient objects based on multimodal feature fusion according to claim 4, characterized in that: The method of step S3 is: Five layers of primary fusion features are obtained through the feature fusion module, and the primary fusion features of adjacent layers and the output features of the decoding block D of the previous layer are sent to the fusion module S; First, the primary fusion features of the two adjacent layers are added together, and then the spatial attention map is obtained through average pooling in the channel dimension. The spatial attention map is used to optimize the features, and then the output of the previous decoding block is added to the current layer. Finally, another spatial attention is used to further optimize the features. The process can be expressed as: ； =IN( + ),i=1,2,3,4; Among them, Upsampling(·) is bilinear upsampling twice, is the output of multimodal information fusion at layer i, is the output of the previous layer decoding module.

6. The method for identifying salient objects based on multimodal feature fusion according to claim 5, characterized in that: The output of the fourth fusion module S jumps backward to the input of each decoding block D and is added to the output of the fusion module S, and finally sent to the decoding block D together.

Images