A salient object detection method for unregistered RGB-T images of UAVs

Abstract

The present invention discloses a salient target detection method for unregistered RGB-T images of drones. The method first obtains the RGB modality and thermal infrared modality data of the scene and performs preprocessing. Secondly, the backbone network is used to extract n-level different scale features of the RGB modality and thermal infrared modality respectively, and modal feature enhancement is performed. Then, the RGB modal features and thermal infrared modal features after modal feature enhancement are subjected to cross-modal feature fusion. Finally, the fused features are subjected to target search decoding to obtain saliency detection results, and are jointly trained in an end-to-end manner. The present invention can help the network better focus on important feature information and improve the performance and accuracy of the network in image processing tasks.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically refers to a salient object detection method for unregistered RGB-T images of unmanned aerial vehicles. Background Art

[0002] With the development of drone technology, drones are widely used in various industries, including environmental monitoring, firefighting and rescue, and power inspection. Various visual tasks based on the drone perspective have been extensively studied, and various methods for detecting salient objects in drone-based images have emerged. Salient object detection in drone-based images is a research task that uses computers to simulate human visual attention mechanisms to segment visually salient objects from visible light images captured by drones. Currently, this task has received widespread attention. Visual attention information can be used in many drone vision tasks, and therefore, salient object detection algorithms based on drone-based images are widely used in various topics, such as drone-based visual tracking, drone-based visual localization, and drone-based behavior recognition. Salient object detection for drones has important research value and significance.

[0003] Although visible light images captured by drones can capture detailed information such as target color and texture, they struggle to provide sufficient information for visual algorithms in challenging environments such as low light and complex backgrounds, significantly impacting the accuracy of salient object detection. With the diversification of imaging technologies and information description methods, imaging information from a variety of sensors is being used in drone-based salient object detection. Among them, thermal infrared images capture the temperature field of the object's surface. Their image quality is insensitive to lighting and unaffected by haze and smog. The integration of visible light and thermal infrared information has proven effective in various vision tasks. Therefore, in the task of detecting salient objects in drone-based images, if the target image quality is poor in visible light, utilizing thermal infrared images to further assist detection can, to a certain extent, overcome the effects of background clutter, poor lighting conditions, and inclement weather, improving salient object detection performance.

[0004] Unlike traditional UAV-based salient object detection methods, which primarily target RGB images or registered RGB-T images, these methods achieve good results. However, RGB images do not provide sufficient effective information in complex environments, and registered RGB-T images can be incompletely registered due to flaws in existing registration algorithms. This reduces the accuracy of salient object detection. Considering the shortcomings of traditional UAV-based salient object detection methods, an increasing number of researchers have begun to investigate salient object detection in unregistered RGB-T images. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the present invention provides a salient object detection method for unregistered RGB-T images of UAVs.

[0006] The method of the present invention comprises the following steps:

[0007] Step (1). Obtain the RGB modality and thermal infrared modality data of the scene and perform preprocessing.

[0008] Step (2). Feature extraction module, through the backbone network, extracts n-level different scale features of RGB modality and thermal infrared modality, respectively, which are recorded as and i=1,...,n.

[0009] Step (3). The modal feature enhancement module performs modal feature enhancement on the RGB modal features and thermal infrared modal features of different scales. The specific method is as follows:

[0010] The RGB modal features and thermal infrared modal features of different scales are average pooled, max pooled and 1x1 convolution are performed along the channel dimension. Then, the global spatial feature map f is generated by splicing and convolution compression. s :

[0011]

[0012] f s =δ(Conv(Cat([S 1 ,...,S n ])))

[0013] in, represents the i-th level RGB modal feature or thermal infrared modal feature, Ave represents the average pooling operation, Max represents the maximum pooling operation, Conv represents the 1x1 convolution operation, Cat represents the splicing operation, and δ represents the Sigmoid activation function. Then, the global spatial feature map is element-wise multiplied with the original feature, and the original feature is added. For the RGB modal feature, a channel attention mechanism is added to emphasize the semantic information:

[0014]

[0015]

[0016] in, represents the i-th level modality feature of the original RGB modality, F t i represents the i-th level modal feature of the original thermal infrared mode, represents the i-th level modality feature of the enhanced RGB modality, F ti' represents the i-th level modal feature of the enhanced thermal infrared mode, f s represents the global spatial feature map, and CA represents the channel attention mechanism.

[0017] Step (4). The cross-modal feature fusion module performs cross-modal feature fusion on the RGB modal features and thermal infrared modal features after modal feature enhancement. The specific method is as follows:

[0018] First, the enhanced RGB modal features and thermal infrared modal features are concatenated and convolved to form an initial feature vector. Then, the average pooling operation and the Softmax activation function are applied in sequence to generate the weights of each modality. and The weight of each modality is multiplied by the modal features to redistribute the weight of each modality fusion:

[0019]

[0020] Next, the reweighted modal features are fused at multiple levels through convolution, and an attention mechanism (CBAM) is introduced in the middle.

[0021]

[0022] Step (5). The target search decoding module performs target search decoding on the fused features to obtain the saliency detection results. The specific method is as follows:

[0023] First, perform a preliminary integration of features of different scales (this operation is not required for features of the nth layer):

[0024]

[0025] in, represents the fused i-th level modal features, represents the modal features after decoding at level i+1, Represents the modal features of the i-th level after preliminary integration.

[0026] Then, based on With the help of position prior P, semantic information is inferred to obtain the target location information

[0027] Next, high-frequency and low-frequency filters are used to enhance valuable information and filter out interference information. The filtered information includes low-frequency information, high-frequency information, and original information. The sum of low-frequency information, high-frequency information, and original information is used as the filtered feature, and combined with convolution to perform the following operation:

[0028]

[0029] Among them, HF(·) is the high-frequency filter function, and LF(·) is the low-frequency filter function. The top two layers use the residual structure to enhance the high-level features:

[0030]

[0031] Here, N(·) is defined as layer normalization, and FFN(·) represents the feedforward network. The remaining scales use a residual structure to achieve optimized decoding of the target. Combined with the channel attention mechanism, a 1×1 convolution kernel is used to generate the saliency detection result. The calculation is as follows:

[0032]

[0033] Step (6). The feature extraction module, modal feature enhancement module, cross-modal feature fusion module and target search decoding module are jointly trained in an end-to-end manner through the loss function.

[0034] The beneficial effects of the present invention are as follows:

[0035] Before fusing the RGB and thermal infrared image modal features, global feature maps for both modalities are calculated. Feature enhancement is then performed on these modal features to reduce image noise and redundant information, improving image quality and accuracy. A dynamic weighting strategy is employed during cross-modal feature fusion to adaptively adjust the ratio of the RGB and thermal infrared image modal features during cross-modal feature fusion. A channel- and spatial-focused attention mechanism is also introduced to help the network better focus on important feature information, improving its performance and accuracy in image processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is the overall network framework diagram of the method of the present invention;

[0037] Figure 2 : is a structural diagram of the modal feature enhancement module in the method of the present invention;

[0038] Figure 3 : is a structural diagram of the cross-modal feature fusion module in the method of the present invention;

[0039] Figure 4 This is a diagram of the architecture of target search decoding in the method of the present invention;

[0040] Figure 5 This is a result diagram of the method of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to the accompanying drawings.

[0042] Figure 1This is the overall network framework of the method of the present invention, a salient object detection method for unregistered RGB-T images of drones, with the following specific steps:

[0043] Step (1). Data acquisition and preprocessing, the specific method is as follows:

[0044] First, the RGB sensor and thermal infrared sensor carried by the drone are used to capture the RGB modality and thermal infrared modality of the same scene at the same time and angle. Then, the bilinear interpolation method is used to resize the captured RGB modality and thermal infrared modality to 352×352. At the same time, the modal data is standardized to make it present a normal distribution, which is conducive to faster model convergence and improves model performance.

[0045] Step (2). Feature extraction module, the specific method is as follows:

[0046] Resnet50 is used as the backbone network to extract 5-level features of different scales of RGB modality and thermal infrared modality, which are respectively recorded as and F t i (i=1,2,3,4,5).

[0047] Step (3). Modal feature enhancement module, the specific method is as follows:

[0048] like Figure 2 As shown in Figure 1, the RGB modal features and thermal infrared modal features of different scales are first average pooled, max pooled and 1x1 convolutional along the channel dimension. Then, the global spatial feature map f is generated by splicing and convolution compression. s :

[0049]

[0050] f s =δ(Conv(Cat([S 1 ,S 2 ,S 3 ,S 4 ,S 5 ])))

[0051] in, The ith level RGB or thermal infrared modal features are represented by Ave, Max, Conv, Cat, and δ, respectively. The global spatial feature map is then element-wise multiplied with the original features to enhance spatial detail. For RGB modal features, a channel attention mechanism is added to emphasize semantic information.

[0052]

[0053] F t i' =(F t i *f s )+F t i

[0054] in, represents the i-th level modality feature of the original RGB modality, F t i represents the i-th level modal feature of the original thermal infrared mode, represents the i-th level modality feature of the enhanced RGB modality, F t i' represents the i-th level modal feature of the enhanced thermal infrared mode, f s represents the global spatial feature map, and CA represents the channel attention mechanism.

[0055] Step (4). Cross-modal feature fusion module. The specific method is as follows:

[0056] like Figure 3 As shown in the figure, the RGB modal features and thermal infrared modal features are first concatenated and convolved to form an initial feature vector. Then, the average pooling operation and the Softmax activation function are applied in sequence to generate the weights of each modality. and The weight of each modality is multiplied by the modal features to redistribute the weight of each modality fusion:

[0057]

[0058] Next, the reweighted modal features are fused at multiple levels through convolution, with a CBAM (Channel-Attention Mechanism) introduced in between. CBAM enhances the performance of convolutional neural networks by introducing channel-attention and spatial-attention mechanisms. The channel-attention mechanism helps the network better learn and utilize feature correlations between channels, improving its ability to extract and utilize key features. The spatial-attention mechanism helps the network better learn and utilize feature correlations between different spatial locations, enhancing its processing of spatial information.

[0059]

[0060] Step (5). Target search decoding module, the specific method is as follows:

[0061] like Figure 4 As shown, first, the features of different scales are preliminarily integrated (this operation is not required for the 5th layer features):

[0062]

[0063] in, represents the fused i-th level modal features, represents the modal features after decoding at level i+1, Represents the modal features of the i-th level after preliminary integration.

[0064] Then, the semantic information is inferred with the help of the position prior P to obtain the target's position information:

[0065]

[0066] Among them, cos(·) represents the cosine similarity attention mechanism, Defined as matrix multiplication, k i ,v i ,q i represents a learnable parameter.

[0067] Next, high-frequency and low-frequency filters are used to enhance valuable information and filter out interference information. The filtered information includes low-frequency information, high-frequency information, and original information. The sum of low-frequency information, high-frequency information, and original information is used as the filtered feature, and combined with convolution to perform the following operation:

[0068]

[0069] Among them, HF(·) is the high-frequency filter function, and LF(·) is the low-frequency filter function. The top two layers use the residual structure to enhance high-level features and avoid the disappearance of high-level features:

[0070]

[0071] Here, N(·) is defined as layer normalization, and FFN(·) represents a feedforward network. The remaining three layers use a residual structure to achieve optimal decoding of the target. Combined with the channel attention mechanism, a 1×1 convolution kernel is used to generate the saliency detection result. The calculation is as follows:

[0072]

[0073] The significant prediction results are as follows Figure 5 The salient target area prediction diagram is shown in FIG.

[0074] Step (6). The feature extraction module, modal feature enhancement module, cross-modal feature fusion module and target search decoding module are jointly trained in an end-to-end manner. The specific method is as follows:

[0075] First, we built a detection network model based on the Pytorch deep learning framework. During the training phase, we used ResNet50 as the backbone network and initialized the parameters using the pre-trained parameters on ImageNet. We set the number of network training rounds to 100 and used the stochastic gradient descent optimizer to optimize the network. The initial weight decay was set to 5×10 -4 The momentum was set to 0.9, and the learning rate was 0.001. The learning rate was reduced to 1 / 10 of its original value every 40 epochs. The model was trained using the training set from the UAV RGB-T 2400 dataset. This training set contains 1200 RGB-T image pairs, all captured by a professional drone (DJI MAVIC 2 Enterprise Advanced) at the same time, from the same angle, and in the same scene. The RGB images are sized at 1920×1080, and the thermal infrared images are sized at 640×512. During data preprocessing, bilinear interpolation was used to resize the input RGB and thermal infrared images to 352×352, and the batch size was set to 4.

Claims

1. A salient object detection method for unregistered RGB-T images of UAVs, characterized by: The following steps are involved: Step 1. Obtain the RGB modality and thermal infrared modality data of the scene and perform preprocessing; Step 2. Extract n-level features of different scales for RGB and thermal infrared modalities through the backbone network. Step 3. Enhance the RGB modal features and thermal infrared modal features at different scales. The specific implementation process is as follows: The RGB modal features and thermal infrared modal features of different scales are average pooled, max pooled and 1x1 convolution are performed along the channel dimension; then, the global spatial feature map f is generated by splicing and convolution compression. s : f s =δ(Conv(Cat([S 1 ,...,S n ]))) in, represents the i-th level RGB modal feature or thermal infrared modal feature, Ave represents the average pooling operation, Max represents the maximum pooling operation, Conv represents the 1x1 convolution operation, Cat represents the splicing operation, and δ represents the Sigmoid activation function. Then, the global spatial feature map is element-wise multiplied with the original feature, and the original feature is added. For the RGB modal feature, a channel attention mechanism is added to emphasize the semantic information: in, represents the i-th level modality feature of the original RGB modality, represents the i-th level modal feature of the original thermal infrared mode, represents the i-th level modal feature of the enhanced RGB modality, represents the i-th level modal feature of the enhanced thermal infrared mode, f s represents the global spatial feature map, and CA represents the channel attention mechanism; Step 4. Perform cross-modal feature fusion on the enhanced RGB modal features and thermal infrared modal features. The specific implementation process of cross-modal feature fusion is as follows: First, the enhanced RGB modal features and thermal infrared modal features are concatenated and convolved to form an initial feature vector; then the average pooling operation and the Softmax activation function are applied in sequence to generate the weights of each modality. and The weight of each modality is multiplied by the modal features to redistribute the weight of each modality fusion: The modal features of the redistributed weights are fused at multiple levels through convolution, and the attention mechanism CBAM is introduced in the middle: Step 5. Perform target search decoding on the fused features to obtain saliency detection results; Step 6. Joint training is performed in an end-to-end manner using the loss function.

2. The salient object detection method for unregistered RGB-T images of UAVs according to claim 1, characterized in that: Specifically, step 1 comprises the following steps: first, using the RGB sensor and thermal infrared sensor carried by the drone to capture the RGB mode and thermal infrared mode of the same scene at the same time and angle, respectively; then, using bilinear interpolation, adjusting the sizes of the captured RGB mode and thermal infrared mode to the same size; and at the same time, normalizing the two modal data to make them present a normal distribution.

3. The salient object detection method for unregistered RGB-T images of UAVs according to claim 2, characterized in that: The target search decoding specific implementation process is as follows: First, the features of different scales are preliminarily integrated, and this operation is not performed on the features of the nth layer: in represents the fused i-th level modal features, represents the modal features after decoding at level i+1, represents the modal features of the i-th level after preliminary integration; Then, the semantic information is inferred with the help of the position prior P to obtain the target's position information: Where cos(·) represents the cosine similarity attention mechanism, Defined as matrix multiplication, k i ,v i ,q i represents a learnable parameter; Use high-frequency and low-frequency filters to filter out interference information; the filtered information includes low-frequency information, high-frequency information, and original information. The sum of low-frequency information, high-frequency information, and original information is used as the filtered feature, and combined with convolution to perform the following operation: HF(·) is a high-frequency filter function, and LF(·) is a low-frequency filter function. The top two layers use a residual structure to enhance high-level features: Where N(·) is defined as layer normalization, FFN(·) represents the feedforward network; the remaining scales achieve optimized decoding of the target through the residual structure, combined with the channel attention mechanism, and finally generate the saliency detection result through a 1×1 convolution operation, which is calculated as follows:

Images