Significant target detection method based on multi-band visual image perception and fusion

Abstract

The application provides a salient target detection method based on multi-band visual image perception and fusion, which comprises the following steps: step 1, constructing a multi-band visual image training sample set and labeling the expanded training sample set; step 2, constructing a twin backbone network model and extracting multi-level feature information; step 3, constructing a multi-scale feature extraction network module and extracting multi-scale feature information from each level of feature information; step 4, constructing an attention mechanism module and outputting visible light attention feature maps and thermal infrared attention feature maps; and step 5, constructing a feature fusion module, taking the visible light attention feature maps and the thermal infrared attention feature maps output in step 4 as inputs, fusing them through addition, multiplication and sequential splicing operations, forming a target feature map of a prediction output, and completing the detection of salient targets. The application fully utilizes multi-band visual image information, can realize target recognition in multi-band visual images, and can also improve the recognition accuracy.

Description

Technical Field

[0001] This invention belongs to the fields of image processing and pattern recognition, and specifically relates to a salient target detection method based on multi-band visual image perception and fusion. Background Technology

[0002] Visual perception and image processing technologies, as a key component of artificial intelligence theory and application, have been widely applied in aerospace, intelligent robotics, intelligent driving and other fields, playing a crucial role in important applications such as spacecraft docking, lunar rover visual navigation, and autonomous vehicles.

[0003] Many visual perception applications rely on simple background environments and good lighting conditions. In these scenarios, the images acquired by the visual sensor typically have simple backgrounds, high pixel counts, and minimal lighting interference, greatly reducing the complexity of subsequent image processing. However, in real-world applications, the working scenarios are often quite complex, all facing the common problem of complex lighting environments. The main reasons for this include: changes in daylight intensity between day and night, variations in the strength of light sources such as artificial lights at night, differences in light intensity during scene transitions, and interference from lighting and shadows. The visual perception problems under complex lighting environments pose a severe challenge to visible light visual recognition, significantly impacting detection accuracy and recognition precision.

[0004] To address the challenge that single-modal image data cannot fully represent the information of objects, resulting in performance indicators that fail to meet the needs of practical applications, multimodal information fusion has emerged. Multimodal information fusion effectively combines different feature information from images acquired by sensors across different spectral bands. This maximizes the utilization of complementary information from different modal image data, removes redundant information, and thus obtains a more diverse and comprehensive representation of objects, improving recognition performance. Summary of the Invention

[0005] The purpose of this invention is to propose a salient target detection method based on multi-band visual image perception and fusion, which utilizes the complementarity of multi-band image information and its coupling recognition mechanism from the perspective of multi-band visual perception fusion. This method aims to solve the problem that a single-band image cannot effectively address the complex illumination changes caused by multiple factors, thereby improving the accuracy of target detection.

[0006] To achieve the above objectives, this invention provides a salient target detection method based on multi-band visual image perception and fusion, comprising: Step 1: Constructing a multi-band visual image training sample set, expanding the training sample set using data augmentation techniques, and labeling the expanded training sample set; Step 2: Constructing a Siamese backbone network model, using visible light images and thermal infrared images from the expanded training sample set as input objects, and extracting multi-level feature information; Step 3: Constructing a multi-scale feature extraction network module, using the multi-level feature information output in Step 2 as input, and extracting multi-scale feature information from each level of feature information; Step 4: Constructing an attention mechanism module, which includes channel attention and spatial attention. Step 3: Using the multi-scale feature information obtained in step 3 as input, calculate the spatial attention features and channel attention features corresponding to visible light and thermal infrared, respectively; multiply the spatial attention features of visible light and thermal infrared to obtain the first product feature map, and multiply the first product feature map with the channel attention features of visible light and thermal infrared, respectively, to output the visible light attention feature map and the thermal infrared attention feature map; Step 5: Construct a feature fusion module, using the visible light attention feature map and the thermal infrared attention feature map output in step 4 as input, and fuse them through addition, multiplication and sequential concatenation operations to form the final target feature map as the prediction output, thus completing the detection of salient targets.

[0007] Preferably, step 1 specifically includes: step 1.1: acquiring pairs of visible light images and thermal infrared images through a multi-source camera to form the training sample set; step 1.2: expanding the acquired training sample set using data augmentation technology, and performing pixel-level annotation on the expanded training sample set.

[0008] Preferably, the twin backbone network model constructed in step 2 includes two identical first backbone network models and second backbone network models; the first backbone network model receives the visible light image X. r ∈R W×H×3 As input, extract multi-level feature information of visible light. The second backbone network model receives thermal infrared images X. t ∈R W×H×3 As input, extract multi-level feature information from thermal infrared radiation. Where W and H represent the width and height of the input image, respectively, and N represents the feature level output by the backbone network model.

[0009] Preferably, each of the backbone network models includes an image segmentation layer, a linear embedding layer, an image patch merging layer, and multiple deep self-attention transform network modules for extracting multi-level feature information. The image segmentation layer segments the input visible light image or thermal infrared image into image patches and inputs them into the linear embedding layer. The linear embedding layer performs a linear transformation on each image patch and outputs a linear feature map. The linear data is used as input to begin feature information extraction at each level. Before feature information extraction at each level, the input linear feature map is sampled by the image patch merging layer to reduce the resolution of the linear feature map, adjust the channel dimension, and form a hierarchical feature map. The adjusted hierarchical feature map is input into the deep self-attention transform network module for feature representation to obtain feature information at each level.

[0010] Preferably, the multi-scale feature extraction network module constructed in step 3 includes four parallel convolutional layers with different dilated convolution rates (d = 1, 3, 5, 7). Each dilated convolutional layer uses a 3×3 convolutional kernel and combines different dilated convolution rates d to obtain multi-scale feature information.

[0011] Preferably, the visible light multi-level feature information and thermal infrared multi-level feature information The input is given to the multi-scale feature extraction network module, and visible light multi-scale feature information is obtained from the multi-scale feature map output at each level. and thermal infrared multi-scale feature information in, CAT indicates sequential splicing.

[0012] Preferably, step 4 includes:

[0013] Step 4.1: Extract multi-scale feature information of visible light through global average pooling. and thermal infrared multi-scale feature information Expand to generate channel feature vectors and Two fully connected layers (f) are used c1 f c2 ) for channel feature vectors and Encode; use a Gaussian activation function to convert the channel feature vectors and Each value is mapped to the interval [0,1] to generate channel weights; the generated channel weights are then mapped to the corresponding input visible light multi-scale feature information. and thermal infrared multi-scale feature information Weighting is performed to obtain the attention features of the visible light channel. and thermal infrared channel attention features Its expression is:

[0014]

[0015]

[0016] Where sigmoid represents the Gaussian activation function, f c This represents a fully connected layer, where W represents the weights and b represents the bias. This indicates that corresponding elements of the feature maps are multiplied together.

[0017] Step 4.2: Use 3×3 convolution kernels to generate visible light multi-scale feature information. and thermal infrared multi-scale feature information Weighting is applied to refine the feature information, generating a refined feature map. A 1×1 convolution kernel is used to weight the refined feature map, obtaining the spatial location information of each unit within the refined feature map. A Gaussian activation function is then used to map each feature value within the refined feature map to the interval [0,1], generating spatial location weights. These spatial location weights are then combined with the input visible light multi-scale feature information. and thermal infrared multi-scale feature information Weighting is performed to obtain the spatial attention features of visible light. and thermal infrared spatial attention features Its expression is:

[0018]

[0019]

[0020] Where sigmoid represents the Gaussian activation function, ReLU represents the non-linear activation function, and W represents the weights. This indicates that corresponding elements of the feature maps are multiplied together.

[0021] Step 4.3: Obtain the and Multiplying them yields the first product feature map, achieving spatial alignment; the first product feature map is then multiplied by... and Multiply to output a visible light attention feature map. Thermal infrared attention feature map Achieve channel calibration; and The expressions are as follows:

[0022]

[0023]

[0024] in, This indicates that corresponding elements of the feature maps are multiplied together.

[0025] Preferably, step 5 includes: Step 5.1: Single-level multimodal feature information fusion, which includes:

[0026] Step 5.1.1: Add pixels one by one and The two are merged to form the first fused feature map;

[0027] Step 5.1.2: Combine the first fused feature map with... and Pixel-by-pixel multiplication is performed to form the second and third product feature maps;

[0028] Step 5.1.3: The second and third product feature maps obtained by multiplication are merged again using a sequential concatenation method to obtain the final single-layer feature maps M of each level. i Among them, the single-layer feature map M i Represented as:

[0029]

[0030] in, This indicates pixel-by-pixel addition. `concat` indicates pixel-by-pixel multiplication, while `concat` indicates sequential concatenation.

[0031] Step 5.2: Layer-by-layer feature information fusion, for each layer of single-layer feature map M obtained in Step 5.1 i The process involves fusing high-dimensional single-layer feature maps into low-dimensional single-layer feature maps; this includes:

[0032] Step 5.2.1: Convert high-dimensional single-layer features Figure 2 Double upsampling;

[0033] Step 5.2.2: Use a 1×1 convolution kernel to change the number of channels in the high-dimensional single-layer feature map so that the high-dimensional feature map obtained after doubling the upsampling has the same dimension as the adjacent low-dimensional feature map;

[0034] Step 5.2.3: Add and fuse the feature maps of adjacent layers pixel by pixel to form the target feature map; specifically:

[0035]

[0036] in, _convolution_indicates pixel-wise addition, conv represents a 1×1 convolution kernel with parameters, and UP represents double upsampling.

[0037] This method also includes step 6: constructing a mixture function as the loss function for optimizing the overall network model, wherein the overall network model includes the Siamese backbone network model, the multi-scale feature extraction network module, the attention mechanism module, and the feature fusion module; optimizing the network model using the stochastic gradient descent algorithm; the expression of the mixture function is:

[0038] L loss =αL bce +βL iou +γL ssim

[0039] Where L bce L iou L ssim , represent the cross-entropy loss function, cross-union ratio loss function, and structural similarity index loss function, respectively; α, β, and γ represent the balance parameters.

[0040] Preferably, the optimization of the overall network model using the stochastic gradient descent algorithm includes: the batch size of each input to the network is 16, the momentum coefficient is 0.9, and the weight decay coefficient is 0.0005; the initial learning rate of the network is 5e-5, and the learning rate is reduced by 1 / 10 for every 10 batches trained, for a total of 300 batches trained.

[0041] In summary, compared with existing technologies, the salient target detection method based on multi-band visual image perception and fusion provided by this invention has the following beneficial effects:

[0042] (1) Using Swing Transformer as the backbone neural network can extract more robust and richer hierarchical features.

[0043] (2) The multi-scale feature extraction module adopts multiple parallel residual dilated convolutions, which improves the network model's ability to perceive multi-scale changes of the target while preserving the original information and without increasing the parameters.

[0044] (3) The attention mechanism fully integrates target information from different modalities, enabling the model to focus on the target itself, improving the network model's comprehensive perception of the target, and enhancing the recognition accuracy.

[0045] (4) The hybrid loss function makes the model more compact in its representation of the target sample features, while enhancing the separability of the target and the background, accelerating the optimization of the network model, and improving the recognition accuracy. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the overall process of the method described in this invention;

[0047] Figure 2This is a diagram of the twin backbone neural network structure used for multi-band visual image feature extraction in this invention;

[0048] Figure 3 This is a schematic diagram of the multi-scale feature extraction module in this invention;

[0049] Figure 4 This is a schematic diagram of the attention mechanism module in this invention;

[0050] Figure 5 This is a schematic diagram of the feature fusion module in this invention. Detailed Implementation

[0051] The following will be combined with the appendix in the embodiments of the present invention. Figure 1 ~Attached Figure 5 The technical solutions, structural features, objectives and effects achieved in the embodiments of the present invention will be described in detail.

[0052] It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, intended only to facilitate and clarify the illustration of the embodiments of the present invention, and are not intended to limit the implementation conditions of the present invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationships, or adjustments in the size, without affecting the effects and objectives achieved by the present invention, should still fall within the scope of the technical content disclosed in the present invention. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only the elements expressly listed, but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0053] This invention provides a salient target detection method based on multi-band visual image perception and fusion, such as... Figure 1 As shown, this salient target detection method includes:

[0054] Step 1: Construct a multi-band visual image training sample set, expand the training sample set using data augmentation techniques, and label the expanded training sample set.

[0055] Step 2: Construct a twin backbone network model, using visible light images and thermal infrared images from the expanded training sample set as input objects, and extract multi-level feature information;

[0056] Step 3: Construct a multi-scale feature extraction network module, using the multi-level feature information output in Step 2 as input, to extract multi-scale feature information from each level of feature information;

[0057] Step 4: Construct an attention mechanism module, which includes channel attention and spatial attention; using the multi-scale feature information of each level obtained in Step 3 as input, calculate the spatial attention features and channel attention features corresponding to visible light and thermal infrared, respectively; multiply the spatial attention features of visible light and the spatial attention features of thermal infrared to obtain a first product feature map, and multiply the first product feature map with the channel attention features of visible light and the channel attention features of thermal infrared, respectively, to output the visible light attention feature map and the thermal infrared attention feature map;

[0058] Step 5: Construct a feature fusion module. Using the visible light attention feature map and thermal infrared attention feature map output in Step 4 as input, fuse them through addition, multiplication and sequential concatenation operations to form the final target feature map as the prediction output, thus completing the detection of salient targets.

[0059] Specifically, step 1 includes: step 1.1: acquiring pairs of visible light images and thermal infrared images through a multi-source camera to form the training sample set; the training sample set includes different light intensities and diverse backgrounds; step 1.2: expanding the acquired training sample set using data augmentation technology, and using Labelme open-source software to perform pixel-level annotation on the expanded training sample set.

[0060] Furthermore, such as Figure 2As shown, the twin backbone network model constructed in step 2 includes two identical first and second backbone network models, used for feature representation of visible light and thermal infrared images, respectively, with parameters shared between them. Each backbone network model is mainly constructed based on a self-attention mechanism, and its structure mainly includes an image segmentation layer, a linear embedding layer, an image patch merging layer, and multiple deep self-attention transform network (Swin Transformer, SwinT) modules for extracting multi-level feature information. The backbone network model includes four stages: The first stage consists of an image segmentation layer, a linear embedding layer, and deep self-attention transform network modules. Specifically, the input image is segmented into image patches, similar to downsampling, with the aim of reducing the resolution of the input image and reducing computational load. Then, each image patch is input into the linear embedding layer, and a linear transformation is performed on each image patch to output linear features. Finally, the linear features are input into the deep self-attention transform network module, which extracts rich semantic features through the self-attention mechanism and multi-head learning mechanism, while ensuring the number of image patches, thus obtaining feature information at each level. The second, third, and fourth stages have similar structures, mainly consisting of an image merging layer and a deep self-attention transform network module. The image patch merging layer downsamples the input features, reducing the feature map resolution and adjusting the channel dimensions to form a hierarchical feature map. Then, the deep self-attention transform network module extracts rich semantic features through self-attention and multi-head learning mechanisms, obtaining hierarchical feature information. Taking visible light images as an example, the detailed functions of each layer are as follows:

[0061] (1) The image segmentation layer segments the input visible light image or thermal infrared image into image blocks and inputs them into the linear embedding layer; in one embodiment, the image segmentation layer uses a convolution kernel of size 4×4×48 and stride 4 to process the input image X r ∈R W×H×3 The image is divided into 4×4 pixel blocks, with no overlap between the blocks.

[0062] (2) Linear Embedding Layer: Each image block undergoes a linear transformation, and a linear feature map is output. This linear data is used as input to begin feature information extraction at each level. Specifically, in one embodiment, the output of the image block layer is linearly transformed through the linear embedding layer, changing the channel data of each pixel from 48 to C, to obtain...

[0063] (3) Image patch merging layer: Before feature information extraction at each level, the input linear feature map is sampled through the image patch merging layer to reduce the resolution of the linear feature map, adjust the channel dimension, and form a hierarchical feature map. Specifically, in one embodiment, the image patch merging layer selects elements at position intervals of 2, splices them into new patches of 4 pixels, and then splices all the new patches together as a whole tensor to expand. After that, a fully connected layer is used to adjust the channel dimension to twice the original value to obtain the hierarchical feature map.

[0064] (4) Deep Self-Attention Transformer (SwinT) Module: The hierarchical feature map output from the image patch merging layer is input into the SwinT module for feature representation. Rich semantic features are extracted through self-attention mechanism and multi-head learning mechanism, while ensuring the number of image patches and obtaining feature information at each level.

[0065] Specifically, in this embodiment, step 2 constructs the aforementioned twin backbone network model, which includes two identical first backbone network models and second backbone network models; the first backbone network model receives the visible light image X. r ∈R W ×H×3 As input, extract multi-level feature information of visible light. The second backbone network model receives thermal infrared images X. t ∈R W×H×3 As input, extract multi-level feature information from thermal infrared radiation. Where W and H represent the width and height of the input image, respectively, and N represents the feature level output by the backbone network model; preferably, N = 4.

[0066] like Figure 3 As shown, the multi-scale feature extraction network module constructed in step 3 includes four parallel convolutional layers with different dilated convolution rates (d = 1, 3, 5, 7). Each dilated convolutional layer employs a 3×3 convolutional kernel and combines different dilation rates (d) to acquire multi-scale feature information. In this embodiment, the output channel dimension of each dilated convolutional layer is 64. A residual-like connection is used to short-connect the input and output, reducing computation while preserving the original information. The features output by each convolutional layer are batch regularized and non-linearly activated. Furthermore, features output by different dilation rates are fused using a sequential concatenation method. The channel dimension of the fused feature map is reduced to 64 using a 1×1 convolutional kernel, and then activated using a non-linear activation function.

[0067] Furthermore, the multi-scale feature extraction network module is applied to multi-level feature information of visible light. and thermal infrared multi-level feature information Specifically, the visible light multi-level feature information and thermal infrared multi-level feature information The input is the multi-scale feature extraction network module, which obtains high-dimensional visible light multi-scale feature information of each level from the multi-scale feature map output at each level. and high-dimensional thermal infrared multi-scale feature information in, CAT indicates sequential splicing.

[0068] like Figure 4 As shown, an attention mechanism module is constructed, which includes channel attention and spatial attention; the output visible light attention feature map and thermal infrared attention feature map in step 4 specifically include steps 4.1, 4.2 and 4.3.

[0069] Step 4.1 is implemented based on the channel attention mechanism, which consists of global average pooling, two fully connected layers, and a Gaussian activation function layer. Global average pooling is used to extract multi-scale feature information of visible light. and thermal infrared multi-scale feature information Expand to generate channel feature vectors and Two fully connected layers (f) are used c1 f c2 ) for channel feature vectors and Encode; use a Gaussian activation function to convert the channel feature vectors and Each value is mapped to the interval [0,1] to generate channel weights; the generated channel weights are then mapped to the corresponding input visible light multi-scale feature information. and thermal infrared multi-scale feature information Weighting is performed to obtain the attention features of the visible light channel. and thermal infrared channel attention features Its expression is:

[0070]

[0071]

[0072] Where sigmoid represents the Gaussian activation function, f c This represents a fully connected layer, where W represents the weights and b represents the bias. This indicates that corresponding elements of the feature maps are multiplied together.

[0073] Furthermore, step 4.2 is implemented based on the spatial attention, which consists of a 3×3 convolutional layer, a 1×1 convolutional layer, and a Gaussian activation function layer. The 3×3 convolutional kernel is used in conjunction with visible light multi-scale feature information. and thermal infrared multi-scale feature information Weighting is applied to refine the feature information, generating a refined feature map. A 1×1 convolution kernel is used to weight the refined feature map, obtaining the spatial location information of each unit within the refined feature map. A Gaussian activation function is then used to map each feature value within the refined feature map to the interval [0,1], generating spatial location weights. These spatial location weights are then combined with the input visible light multi-scale feature information. and thermal infrared multi-scale feature information Weighting is performed to obtain the spatial attention features of visible light. and thermal infrared spatial attention features Its expression is:

[0074]

[0075]

[0076] Where sigmoid represents the Gaussian activation function, ReLU represents the non-linear activation function, W represents the weights, and b represents the bias. This indicates that corresponding elements of the feature maps are multiplied together.

[0077] Furthermore, step 4.3 includes: obtaining... and Multiplying them yields the first product feature map, achieving spatial alignment; the first product feature map is then multiplied by... and Multiply to output a visible light attention feature map. Thermal infrared attention feature map Achieve channel calibration; and The expressions are as follows:

[0078]

[0079]

[0080] in, This indicates that corresponding elements of the feature maps are multiplied together.

[0081] like Figure 5 As shown, Figure 5This is a schematic diagram of the feature fusion module. Based on this feature fusion module, the visible light attention feature map and the thermal infrared attention feature map output in step 4 are used as inputs. Step 5 includes: step 5.1: single-level multimodal feature information fusion, and step 5.2: hierarchical feature information fusion. The target feature map is formed through steps 5.1 and 5.2 to complete the detection of salient targets.

[0082] Specifically, step 5.1 includes the following steps:

[0083] Step 5.1.1: Add pixels one by one and The two are merged to form the first fused feature map;

[0084] Step 5.1.2: Combine the first fused feature map with... and Pixel-by-pixel multiplication is performed to form the second and third product feature maps;

[0085] Step 5.1.3: The second and third product feature maps obtained by multiplication are merged again using a sequential concatenation method to obtain the final single-layer feature maps M of each level. i Among them, the single-layer feature map M i Represented as:

[0086]

[0087] in, This indicates pixel-by-pixel addition. This indicates pixel-by-pixel multiplication, while Concat indicates sequential concatenation.

[0088] Specifically, the layer-by-layer feature information fusion described in step 5.2 is based on the single-layer feature maps M obtained in step 5.1. i The process involves fusing high-dimensional single-layer feature maps into low-dimensional single-layer feature maps; this includes:

[0089] Step 5.2.1: Convert high-dimensional single-layer features Figure 2 Double upsampling;

[0090] Step 5.2.2: Use a 1×1 convolution kernel to change the number of channels in the high-dimensional single-layer feature map so that the high-dimensional feature map obtained after doubling the upsampling has the same dimension as the adjacent low-dimensional feature map;

[0091] Step 5.2.3: Merge the feature maps of adjacent layers pixel by pixel to form the target feature map, specifically as follows:

[0092]

[0093] in, _convolution_indicates pixel-wise addition, conv represents a 1×1 convolution kernel with parameters, and UP represents double upsampling.

[0094] To obtain a model with stronger robustness and generalization, the salient object detection method provided by this invention further includes: Step 6: Constructing a mixture function as the loss function for optimizing the overall network model, wherein the overall network model includes the Siamese backbone network model, the multi-scale feature extraction network module, the attention mechanism module, and the feature fusion module; optimizing the network model using the stochastic gradient descent algorithm; the expression of the mixture function is:

[0095] L loss =αL bce +βL iou +γL ssim

[0096] Where L bce L iou L ssim , represent the cross-entropy loss function, cross-union ratio loss function, and structural similarity index loss function, respectively; α, β, and γ represent the balance parameters.

[0097] The cross-entropy loss function BCE is used to reflect the difference in probability distribution between the predicted value and the true value, specifically:

[0098] L bce =-∑T r logP r -∑(1-T r log(1-P) r )

[0099] The Cross-Union Ratio (IoU) loss is used to measure the similarity between two sets, specifically:

[0100]

[0101] Among them, T r ∈{0,1} represents the true value of pixel r, P r ∈{0,1} represents the probability value that pixel r is predicted as the target.

[0102] The structural similarity index SSIM can capture structural information in images, guiding network models to learn the ground-value structural information of salient targets. Assume p = {p i :i=1,···,N 2} and t={t i :i=1,···,N 2} are the pixel sets of two corresponding-sized (size: N×N) image regions cropped from the predicted probability map P and the true value T, respectively. The SSIM of p and t is represented as:

[0103]

[0104] Where v p v t and These are the mean and variance of p and t, respectively, and σ pt These are the corresponding covariances; C1 and C2 are each set to 0.01. 2 and 0.0 3 To avoid being divided by zero.

[0105] Furthermore, using the designed hybrid function as the loss function, the optimization of the overall network model using the stochastic gradient descent algorithm includes: the batch size of each input to the network is 16, the momentum coefficient is 0.9, and the weight decay coefficient is 0.0005; the initial learning rate of the network is 5e-5, and the learning rate is reduced by 1 / 10 for every 10 batches trained, for a total of 300 batches trained to optimize the network model.

[0106] In summary, compared with the prior art, the salient target detection method based on multi-band visual image perception and fusion provided by the present invention makes full use of multi-band visual image information, which not only realizes target recognition in multi-band visual images, but also improves the recognition accuracy.

[0107] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A salient target detection method based on multi-band visual image perception and fusion, characterized in that, include: Step 1: Construct a multi-band visual image training sample set, expand the training sample set using data augmentation techniques, and label the expanded training sample set. Step 2: Construct a twin backbone network model, using visible light images and thermal infrared images from the expanded training sample set as input objects, to extract multi-level feature information; wherein, the constructed twin backbone network model includes two identical first backbone network models and second backbone network models; the first backbone network model and the second backbone network model respectively include an image segmentation layer, a linear embedding layer, an image patch merging layer, and multiple deep self-attention transformation network modules, used to extract multi-level feature information; Step 3: Construct a multi-scale feature extraction network module, using the multi-level feature information output from Step 2 as input, to extract multi-scale feature information from each level of feature information; wherein, the constructed multi-scale feature extraction network module includes four parallel convolutional layers with different dilation rates. Each dilated convolutional layer uses a 3×3 convolutional kernel and combines different dilation rates d to obtain multi-scale feature information. The different dilation rates d... ; Step 4: Construct an attention mechanism module, which includes channel attention and spatial attention; using the multi-scale feature information of each level obtained in Step 3 as input, calculate the spatial attention features and channel attention features corresponding to visible light and thermal infrared, respectively; multiply the spatial attention features of visible light and the spatial attention features of thermal infrared to obtain a first product feature map, and multiply the first product feature map with the channel attention features of visible light and the channel attention features of thermal infrared, respectively, to output the visible light attention feature map and the thermal infrared attention feature map; Step 5: Construct a feature fusion module. Using the visible light attention feature map and thermal infrared attention feature map output in Step 4 as input, fuse them through addition, multiplication and sequential concatenation operations to form the final target feature map as the prediction output, thus completing the detection of salient targets. Step 6: Construct a mixture function as the loss function for optimizing the overall network model. The overall network model includes the Siamese backbone network model, the multi-scale feature extraction network module, the attention mechanism module, and the feature fusion module. Optimize the network model using the stochastic gradient descent algorithm. The expression for the mixture function is: in L bce , L iou , L ssim , respectively represent the cross-entropy loss function, the cross-union ratio loss function, and the structural similarity index loss function; , , This represents the equilibrium parameters.

2. The salient target detection method based on multi-band visual image perception and fusion as described in claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Acquire pairs of visible light images and thermal infrared images using a multi-source camera to form the training sample set; Step 1.2: Use data augmentation techniques to expand the collected training sample set, and then perform pixel-level annotation on the expanded training sample set.

3. The salient target detection method based on multi-band visual image perception and fusion as described in claim 2, characterized in that, The first backbone network model receives visible light images. As input, extract multi-level feature information of visible light. ; The second backbone network model receives thermal infrared images. As input, extract multi-level feature information from thermal infrared radiation. ; in, W and H These represent the width and height of the input image, respectively. N This represents the feature level output by the backbone network model.

4. The salient target detection method based on multi-band visual image perception and fusion as described in claim 3, characterized in that, The image segmentation layer cuts the input visible light image or thermal infrared image into individual image blocks and inputs them into the linear embedding layer; The linear embedding layer performs a linear transformation on each image patch and outputs a linear feature map; the linear data is then used as input to begin feature information extraction at each level. Before feature information extraction at each level, the input linear feature map is downsampled through the image block merging layer to reduce the resolution of the linear feature map, adjust the channel dimension, and form a hierarchical feature map. The adjusted hierarchical feature map is input into the deep self-attention transformation network module for feature representation to obtain feature information of each level.

5. The salient target detection method based on multi-band visual image perception and fusion as described in claim 3, characterized in that, Visible light multi-level feature information and thermal infrared multi-level feature information The input is given to the multi-scale feature extraction network module, and visible light multi-scale feature information is obtained from the multi-scale feature map output at each level. and thermal infrared multi-scale feature information ;in, , , CAT This indicates sequential splicing.

6. The salient target detection method based on multi-band visual image perception and fusion as described in claim 5, characterized in that, Step 4 includes: Step 4.1: Extract multi-scale feature information of visible light through global average pooling. and thermal infrared multi-scale feature information Expand to generate channel feature vectors and Two fully connected layers are used ( f c1 , f c2 ) for channel feature vectors and Encode; use a Gaussian activation function to convert the channel feature vectors and Each value is mapped to the interval [0, 1] to generate channel weights; the generated channel weights are then mapped to the corresponding input visible light multi-scale feature information. and thermal infrared multi-scale feature information Weighting is performed to obtain the attention features of the visible light channel. and thermal infrared channel attention features Its expression is: in, sigmoid Represents the Gaussian activation function. f c Indicates a fully connected layer. W Indicates weight, b Indicates bias. This indicates that corresponding elements of the feature maps are multiplied together. Step 4.2: Use 3×3 convolution kernels to generate visible light multi-scale feature information. and thermal infrared multi-scale feature information Weighting is applied to refine the feature information, generating a refined feature map. A 1×1 convolution kernel is used to weight the refined feature map, obtaining the spatial location information of each unit within the refined feature map. A Gaussian activation function is then used to map each feature value within the refined feature map to the interval [0, 1], generating spatial location weights. These spatial location weights are then combined with the input visible light multi-scale feature information. and thermal infrared multi-scale feature information Weighting is performed to obtain the spatial attention features of visible light. and thermal infrared spatial attention features Its expression is: in, sigmoid ReLU represents the Gaussian activation function, and ReLU represents the nonlinear activation function. W Indicates weight, This indicates that corresponding elements of the feature maps are multiplied together. Step 4.3: Obtain the and Multiplying them yields the first product feature map, achieving spatial alignment; the first product feature map is then multiplied by... and Multiply to output a visible light attention feature map. Thermal infrared attention feature map To achieve channel calibration; and The expressions are as follows: in, This indicates that corresponding elements of the feature maps are multiplied together.

7. The salient target detection method based on multi-band visual image perception and fusion as described in claim 6, characterized in that, Step 5 includes: Step 5.1: Single-level multimodal feature information fusion, which includes: Step 5.1.1: Add pixels one by one and The two are merged to form the first fused feature map; Step 5.1.2: Combine the first fused feature map with... and Pixel-by-pixel multiplication is performed to form the second and third product feature maps; Step 5.1.3: The second and third product feature maps obtained by multiplication are merged again using a sequential concatenation method to obtain the final single-layer feature maps of each level. M i Among them, single-layer feature maps M i Represented as: in, This indicates pixel-by-pixel addition. This indicates pixel-by-pixel multiplication. Concat Indicates sequential splicing; Step 5.2: Layer-by-layer feature information fusion, for each layer of single-layer feature maps obtained in Step 5.1 M i The process involves fusing high-dimensional single-layer feature maps into low-dimensional single-layer feature maps; this includes: Step 5.2.1: Upsample the high-dimensional single-layer feature map by a factor of two; Step 5.2.2: Use 1 1. The convolution kernel changes the number of channels in a high-dimensional single-layer feature map, so that the high-dimensional feature map obtained after doubling the upsampling has the same dimension as the adjacent low-dimensional feature map. Step 5.2.3: Add and fuse the feature maps of adjacent layers pixel by pixel to form the target feature map; specifically: in, This indicates pixel-by-pixel addition. conv Indicates 1 with parameters 1 convolution kernel, UP This indicates a double upsampling.

8. The salient target detection method based on multi-band visual image perception and fusion as described in claim 1, characterized in that, The optimization of the overall network model using the stochastic gradient descent algorithm includes: a batch size of 16 for each input to the network, a momentum coefficient of 0.9, and a weight decay coefficient of 0.0005; an initial learning rate of 5e-5, with the learning rate decreasing by 1 / 10 for every 10 batches trained, for a total of 300 batches trained.

Images