A salient object detection method based on a hybrid spectral attention U-shaped neural network

Abstract

The application discloses a kind of based on mixed spectral attention U-type neural network's saliency target detection method, belong to hyperspectral saliency target detection field.The method is first to the original hyperspectral image is preprocessed, obtains dimensionality reduction spectral image as main mode, and calculates generation spectral saliency and RGB spectral mapping as supplementary mode;Subsequently, construct based on dynamic sparse double-layer routing self-attention mechanism's U-type coding-decoding network, utilize multimodal spectral mixed frequency attention mechanism integration main, supplementary mode feature, through space-spectral driven feature fusion jump connection enhancement space and spectral feature's availability, and generate saliency prediction graph in each decoding stage and carry out dense supervision.The application effectively solves the problem that effective information loss, spectral feature utilization rate is low and target contour distinguishes difficult in prior art in spectral redundancy, realizes high-precision, high-robustness saliency target detection.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral salient target detection, and more specifically to a salient target detection method based on a hybrid spectral attention U-shaped neural network. Background Technology

[0002] Saliency target detection aims to automatically identify the most critical foreground objects in a scene, and it is of great significance as a fundamental preprocessing step in computer vision research. Hyperspectral saliency target detection leverages hyperspectral images, which contain more spectral information than traditional RGB images, to achieve high-precision detection in challenging scenes such as similar foreground and background, high dynamic range, and complex backgrounds. Hyperspectral images can more accurately represent objects through spectral curves, but existing hyperspectral saliency target detection methods have problems in handling spectral redundancy, effectively utilizing spectral features, and accurately distinguishing target contours.

[0003] Traditional methods often struggle to effectively utilize spectral information, easily losing valuable information during spectral redundancy removal and exhibiting insufficient ability to distinguish target contours. While deep learning-based methods have made significant progress, existing networks typically treat each band as an independent channel when processing hyperspectral images, ignoring the inherent correlations along the spectral dimension, resulting in low utilization of spectral features. Furthermore, simple downsampling operations cause spatial information loss, further impacting detection accuracy. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a salient target detection method based on a hybrid spectral attention U-shaped neural network, so as to solve the problems of spectral redundancy leading to loss of effective information, low utilization of spectral features, and difficulty in accurately distinguishing target contours in the prior art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A salient target detection method based on a hybrid spectral attention U-shaped neural network includes the following steps: S1: Acquire a hyperspectral image and preprocess it to obtain a dimension-reduced spectral image, a spectral saliency image, and an RGB spectral mapping image; wherein, the dimension-reduced spectral image is used as the main mode input, and the spectral saliency image and the RGB spectral mapping image are stitched together to serve as the supplementary mode input; S2: Construct a hybrid spectral attention U-shaped deep neural network, which includes a two-stream encoder, a single-stream decoder, a bottleneck layer, and skip connections. The two-stream encoder is used to extract deep features of the main mode and the complementary mode, respectively. The bottleneck layer is located between the two-stream encoder and the single-stream decoder and is used to integrate the features of the two modes. The single-stream decoder is used to recover the feature space resolution layer by layer and generate a saliency prediction map. The skip connections are used to connect the features of corresponding levels in the two-stream encoder and the single-stream decoder. S3: Input the reduced spectral image and the supplementary modality input image into the hybrid spectral attention U-shaped deep neural network for training. Use the spectral saliency mixture loss function to intensively supervise the network and optimize the network parameters until the network converges. S4: The hyperspectral image to be detected is preprocessed in step S1 and then input into the trained neural network to obtain a saliency prediction map.

[0006] In one possible embodiment, step S1 specifically includes the following process: The original hyperspectral image undergoes image preprocessing including image correction, registration, enhancement, and scaling. The preprocessed image is then uniformly sampled along the channel dimension to obtain a 50-channel dimensionality-reduced image. The center-periphery spectral contrast of the hyperspectral image is calculated to obtain the 3-channel spectral saliency highlighting target discrimination. The RGB spatial dimension of the image is transformed to CIE XYZ spatial coordinates using a matrix to obtain a 3-channel RGB spectral mapping highlighting spatial structural details. The dimensionality-reduced spectral image is used as the main modality input of the method, while the spectral saliency after feature stitching and the RGB spectral mapping are used as the optional modality inputs.

[0007] In one possible embodiment, the dual-stream encoder includes a first branch and a second branch with identical structures. Each branch is divided into three stages. Each stage contains two cascaded two-layer routing blocks and a patch merging layer composed of 3×3 convolutions. Additionally, the first stage includes a patch embedding layer composed of two 3×3 convolutional layers to segment the input feature map into overlapping image patches and embed them into a feature projection with a channel dimension of 64, facilitating subsequent attention calculation. The dual-stream encoder has two branches: the main modality branch encodes the dimensionality-reduced spectral image. To obtain complete spectral depth features By supplementing the modal branch coding spectral saliency and the RGB spectral mapping S,M∈RH×W×3, deep features of high and low frequency spectral components are obtained. The process can be formalized as follows:

[0008]

[0009] In the formula, This indicates an encoder with parameter φ. This represents an encoder with parameter θ. The feature concatenation is represented by the two branches. The features encoded by the two branches converge at the end of the encoder and are integrated by the multimodal spectral mixing frequency attention mechanism to obtain the spectral coding features that achieve the supplementation of high and low frequency components of the spectrum. Finally, the spectral coding features are input into the bottleneck layer.

[0010] In one possible embodiment, the single-stream decoder operates independently in three stages, with a structure similar to that of the decoder. Each stage consists of a patch expanding layer composed of 3×3 convolutional layers and two cascaded two-layer routing blocks. The patch expanding layer increases the resolution of the input feature map by a factor of 2 and reduces the feature dimension by half. The last patch expanding layer performs a 4x upsampling to restore the decoded features to the initial spatial resolution and generates the final pixel-level saliency prediction through linear projection. In addition, the decoder generates saliency predictions and performs dense supervision at each stage, and simultaneously inputs the generated saliency predictions and decoded features into skip connections for optimization layer by layer.

[0011] In one possible embodiment, the two-layer routing block consists of a 3×3 depthwise convolution, two normalization layers, a two-layer routing self-attention layer, three residual connections, and a two-layer multilayer perceptron; its working principle can be formulaically described as follows:

[0012]

[0013]

[0014] In the formula, , , , and These represent depthwise convolution, layer normalization layer, two-layer routing self-attention, and multilayer perceptron, respectively. , and These correspond to the output features of depthwise convolution, double-layer routing self-attention, and multilayer perceptron in the first double-layer routing block, respectively.

[0015] The dual-layer routing self-attention layer is implemented based on a dynamic sparse dual-layer routing self-attention mechanism, which includes, in sequence, region partitioning and input projection, region-to-region routing, and token-to-token attention.

[0016] Region partitioning and input projection, through two-dimensional input features Dividing the region into S × S non-overlapping regions, the feature dimension of each region can be obtained as follows: The characteristics obtained The query tensor, key tensor, and numerical tensor can be obtained through linear projection. : Q=XrWq,K=XrWk,V=XrWv In the formula, These are the weighted projections of the query tensor, the key tensor, and the numerical tensor, respectively. For region-to-region routing, the region-level query tensor and key tensor are first obtained by calculating the average values ​​of Q and K for each region. Then, the adjacency matrix Ar between regions is obtained by applying Qr and Kr. The adjacency matrix is ​​used to query the semantic relevance between two regions, and finally, by using row-by-row top... The k operator retains only the top k most relevant regions for each query region, deriving the routing index matrix. The index matrix contains k indices representing the most relevant regions in each region, and the overall process can be formalized as follows:

[0017]

[0018] In the formula, represent The transpose of the matrix, Represents top row by row k operator; Token-to-token attention, since the routing regions may be scattered across the entire feature map, requires collecting the key tensors and numerical tensors of the routing regions. Then, fine-grained token-to-token attention is applied to the aggregated key-value tensors using an index matrix. The overall process can be described as follows:

[0019]

[0020] In the formula, Indicates an aggregation operation. This represents the Softmax function. It is the aggregated key-value tensor. This indicates that the local context enhancement function is implemented by a depthwise convolution with a kernel size of 5.

[0021] In one possible embodiment, the multimodal spectral mixing frequency attention mechanism aims to promote the comprehensive interactive fusion of main modal features and complementary modal features to obtain complete and rich spectral coding features with high and low frequency components. This mechanism consists of two core parts: a high-frequency attention head and a low-frequency attention head. The high-frequency head promotes the interaction between the main modal features and the high-frequency edge features in the complementary modal features through cross-attention interaction, thereby supplementing the high-frequency edge components. The low-frequency head applies self-attention interaction to the complementary modal features to extract saliency features to enhance the spectrally significant low-frequency components within the features. Finally, the output features of the high-frequency and low-frequency heads are integrated to achieve the correction and supplementation of low-frequency significant components of the main modal coding features under the constraint of enhanced saliency information. The specific principle of this mechanism is as follows: The primary mode coding feature (FHSI) is directly input into the high-frequency head, and the supplementary mode coding feature (FS) is used. M is uniformly divided into two parts along the channel dimension CS and then input into the high-frequency head and low-frequency head respectively. The two sets of features after segmentation are: This mechanism uses neighborhood attention to achieve attentional interaction, and its principle can be formulated as follows:

[0022] In the formula, Represents the pixel neighborhood at position (i,j). It is a relative position offset. It is the Sigmoid activation function; the high-frequency attention head of this mechanism obtains the output features. Low-frequency attention head obtains output features And the final output hybrid frequency decoding feature obtained after frequency fusion. The process is as follows:

[0023]

[0024]

[0025] In the formula, It is the ReLU activation function. This represents a 1×1 convolution.

[0026] The space-spectral driven feature fusion skip connection aims to integrate the input two-modal encoded features, decoded features, and lower-level saliency predictions through a combination of space-spectral attention. This effectively compensates for the spatial information loss caused by encoder downsampling and significantly enhances the network's spectral feature utilization. The core of this skip connection consists of two sub-modules: spatial attention and spectral attention. The spatial attention submodule first captures the spatial details of the input features and generates spatial attention weights. After multiplying with the input feature F1, the output feature F2 is obtained as follows:

[0027]

[0028] In the formula, For element-wise multiplication, It is the Sigmoid activation function. Represents the Softmax function, spatial query tensor and spatial key tensor The result was obtained through F1 processing:

[0029]

[0030] In the formula, It is global average pooling. It is the ReLU activation function. This represents a 1×1 convolution. This represents a multilayer perceptron.

[0031] The spectral attention submodule then extracts the spectral information of the input features and generates empty spectral attention weights. The upper-level decoded features are obtained by multiplying them with themselves to obtain the final output. as follows:

[0032]

[0033] In the formula, Represents a 7×7 convolution; spectral query tensor and spectral key tensor pass The results obtained after processing are as follows:

[0034]

[0035] In the formula, and It consists of 7×7 convolutions, batch normalized layers, and ReLU activation functions.

[0036] The model employs a spectral saliency hybrid loss function, which provides intensive supervision during training by calculating the deviation between the saliency prediction image Y and the corresponding ground truth image T. This loss function consists of three targeted component loss functions: binary cross-entropy loss (BCE), structural similarity index (SSIM), and intersection-over-union ratio (IOU), which are defined as follows:

[0037] In the formula, , and Representing the binary cross-entropy loss, structural similarity index, and intersection-union ratio (IU), respectively; the overall loss of the model can be expressed as:

[0038] In the formula, These are intermediate significance prediction plots for each level. For the final significance prediction, Represents the corresponding truth image. This represents the normalized weight.

[0039] Compared with the prior art, the present invention has the following advantages: 1. This invention adopts a deep learning-based network architecture, combining convolutional neural networks and self-attention mechanisms, to achieve intelligent, efficient, and accurate salient object detection.

[0040] 2. This invention introduces high and low frequency component features of the spectrum, uses multimodal information fusion technology, and designs a spectral dimension attention mechanism to achieve effective hyperspectral image processing and use with redundancy elimination, information integrity, and high utilization rate.

[0041] 3. The method proposed in this invention has strong robustness and is suitable for challenging scenarios such as similar foreground and background, high dynamic range, complex background, and small target. At the same time, the saliency prediction map generated by this method has accurate positioning, complete target, clear outline, and good visualization effect. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0043] Figure 1 This is a flowchart illustrating a salient target detection method based on a hybrid spectral attention U-shaped neural network according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the hybrid spectral attention U-shaped neural network and key components according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of the two-layer routing self-attention mechanism in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of a two-layer routing block according to an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0045] See Figure 1 This invention provides an embodiment of a salient target detection method based on a hybrid spectral attention U-shaped neural network, which includes the following steps: S1, Image processing generates the input dimensionality-reduced spectral image, spectral significance, and RGB spectral mapping; The original hyperspectral image undergoes image preprocessing including image correction, registration, enhancement, and scaling. The preprocessed image is then uniformly sampled along the channel dimension to obtain a 50-channel dimensionality-reduced image. The center-periphery spectral contrast of the hyperspectral image is calculated to obtain the 3-channel spectral saliency highlighting target discrimination. The RGB spatial dimension of the image is transformed to CIE XYZ spatial coordinates using a matrix to obtain a 3-channel RGB spectral mapping highlighting spatial structural details. The dimensionality-reduced spectral image is used as the main modality input of the method, while the spectral saliency after feature stitching and the RGB spectral mapping are used as the optional modality inputs.

[0046] S2, Training a hybrid spectral attention U-shaped deep neural network; A hybrid spectral attention U-shaped deep neural network was trained, which combined a convolutional neural network architecture and a self-attention mechanism. Relying on a dynamic and sparse two-layer routing self-attention mechanism, it effectively extracted local and global semantic information from hyperspectral images. Combined with a layer-by-layer encoder-decoder network architecture, it obtained accurate saliency prediction results. The network is based on a dynamic sparse two-layer routing self-attention mechanism, constructing a two-layer routing block as the basic module. This mechanism consists of three parts: Region Partition and Input Projection, Region-to-Region Routing, and Token-to-Token Attention. The principle is explained in [link to documentation]. Figure 3 The specific principles of the three parts are as follows: Region partitioning and input projection, through two-dimensional input features Dividing the region into S × S non-overlapping regions, the feature dimension of each region can be obtained as follows: The characteristics obtained The query tensor, key tensor, and numerical tensor can be obtained through linear projection. : Q=XrWq,K=XrWk,V=XrWv In the formula, These are the weighted projections of the query tensor, the key tensor, and the numerical tensor, respectively. For region-to-region routing, the region-level query tensor and key tensor are first obtained by calculating the average values ​​of Q and K for each region. Then, the adjacency matrix Ar between regions is obtained by applying Qr and Kr. The adjacency matrix is ​​used to query the semantic relevance between two regions, and finally, by using row-by-row top... The k operator retains only the top k most relevant regions for each query region, deriving the routing index matrix. The index matrix contains k indices representing the most relevant regions in each region, and the overall process can be formalized as follows:

[0047]

[0048] In the formula, represent The transpose of the matrix, Represents top row by row k operator; Token-to-token attention, since the routing regions may be scattered across the entire feature map, requires collecting the key tensors and numerical tensors of the routing regions. Then, fine-grained token-to-token attention is applied to the aggregated key-value tensors using an index matrix. The overall process can be described as follows:

[0049]

[0050] In the formula, Indicates an aggregation operation. This represents the Softmax function. It is the aggregated key-value tensor. This indicates that the local context enhancement function is implemented by a depthwise convolution with a kernel size of 5.

[0051] The two-layer routing block consists of a 3×3 depthwise convolution, two normalization layers, a two-layer routing self-attention layer, three residual connections, and a two-layer multilayer perceptron (MLP). Its structure is described in [link to architecture]. Figure 4 The working principle of this module can be described in a formulaic way as follows:

[0052]

[0053]

[0054] In the formula, , , , and These represent depthwise convolution, layer normalization layer, two-layer routing self-attention, and multilayer perceptron, respectively. , and These correspond to the output features of depthwise convolution, double-layer routing self-attention, and multilayer perceptron in the first double-layer routing block, respectively.

[0055] The network constructs a U-shaped dual-stream coding-single-stream decoding basic architecture by combining basic modules and convolutional layers consisting of image patch embedding layers, image patch merging layers, and image patch expansion layers. See the structure below. Figure 2 (a) The two branches of the dual-stream encoder encode the main modal input and the supplementary modal input respectively. After the image block embedding is completed, the channel dimension of the input features of the two branches is adjusted to 64. Then, after three-stage feature extraction and 2x downsampling, the features are passed down. The full interactive fusion is completed in the multimodal spectral mixing frequency attention mechanism at the bottom to obtain complete and rich spectral coding features of high and low frequency components. The single-stream decoder first passes the features up through three-stage 2x upsampling and feature extraction, generates saliency predictions and performs supervision in stages, and finally generates the final pixel-level saliency predictions through 4x upsampling and linear projection. The bottom bottleneck layer maintains the spatial resolution and channel dimension of the input coded features. After feature integration is completed, the features are input into the decoder.

[0056] In implementation, the network can be divided into 7 stages: stages 1-3 are encoders, stage 4 is the bottleneck layer, and stages 5-7 are decoders. The encoder of stage i is connected to the corresponding decoder of stage 8-i through a spatial-spectral driven feature fusion skip connection. The skip connection integrates the coding features of the two branches of the encoder, the decoding features of the decoder, and the generated saliency predictions, which are then input into the decoder of stage 9-i as upper-layer decoding features.

[0057] The saliency prediction is a probabilistic prediction where pixel values ​​range from 0 to 1. Each pixel value represents the probability that the pixel is the target, corresponding to a binarized ground truth image. The saliency prediction is generated in three steps: feature upsampling, which upsamples features to the same spatial resolution as the ground truth image for easier evaluation and supervision; linear projection, implemented through a 1×1 convolutional layer, transforming the channel dimension to 1; and normalization, using the sigmoid activation function to ensure that each pixel value is between 0 and 1.

[0058] To ensure the effectiveness of network training, the loss between the generated saliency predictions and their corresponding ground truth images is calculated, and the model is subjected to supervised optimization. In implementation, a spectral saliency mixture loss function is used for co-training. By calculating the deviation between the saliency prediction image Y and its corresponding ground truth image T, the training process of the model is intensively supervised. This loss function consists of three targeted component loss functions: binary cross-entropy loss (BCE), structural similarity index (SSIM), and intersection-over-union ratio (IOU), which are defined as follows:

[0059] In the formula, , and Representing the binary cross-entropy loss, structural similarity index, and intersection-union ratio (IU), respectively; the overall loss of the model can be expressed as:

[0060] In the formula, These are intermediate significance prediction plots for each level. For the final significance prediction, Represents the corresponding truth image. This represents the normalized weight.

[0061] S3, designing a multimodal spectral mixing frequency attention mechanism and a space-spectral driven feature fusion skip connection insertion network: The multimodal spectral mixing frequency attention mechanism, located at the bottom of the two-stream encoder, aims to promote the comprehensive interactive fusion of primary and complementary modal features to obtain complete and rich spectral coding features of high and low frequency components. Its structure is described in [reference needed]. Figure 2 (b) This mechanism consists of two core parts: a high-frequency attention head and a low-frequency attention head. The high-frequency head promotes the interaction between the main modality features and the high-frequency edge features in the complementary modality through cross-attention interaction, thereby achieving the supplementation of high-frequency edge components. The low-frequency head applies self-attention interaction with the complementary modality features to extract saliency features to enhance the spectrally significant low-frequency components within the features. Finally, the output features of the high-frequency head and the low-frequency head are integrated to achieve the correction and supplementation of low-frequency significant components of the main modality coding features under the constraint of enhanced saliency information. The specific principle of this mechanism is as follows: The primary mode coding feature (FHSI) is directly input into the high-frequency head, and the supplementary mode coding feature (FS) is used. M is uniformly divided into two parts along the channel dimension CS and then input into the high-frequency head and low-frequency head respectively. The two sets of features after segmentation are: This mechanism uses neighborhood attention to achieve attentional interaction, and its principle can be formulated as follows:

[0062] In the formula, Represents the pixel neighborhood at position (i,j). It is a relative position offset. It is the Sigmoid activation function; the high-frequency attention head of this mechanism obtains the output features. Low-frequency attention head obtains output features And the final output hybrid frequency decoding feature obtained after frequency fusion. The process is as follows:

[0063]

[0064]

[0065] In the formula, It is the ReLU activation function. This represents a 1×1 convolution.

[0066] The spatial-spectral driven feature fusion skip connection connects the dual-stream decoded features in the encoder with the corresponding decoded features and saliency predictions in the decoder. It aims to integrate the input two-modal encoded features, decoded features, and lower-level saliency predictions through spatial-spectral attention combination. Its structure is described in [reference needed]. Figure 2 (c) This skip connection effectively compensates for the spatial information loss caused by encoder downsampling and significantly enhances the utilization rate of the network's spectral features; the core of this skip connection lies in two parts: spatial attention and spectral attention sub-modules. The spatial attention submodule first captures the spatial details of the input features and generates spatial attention weights. After multiplying with the input feature F1, the output feature F2 is obtained as follows:

[0067]

[0068] In the formula, For element-wise multiplication, It is the Sigmoid activation function. Represents the Softmax function, spatial query tensor and spatial key tensor The result was obtained through F1 processing:

[0069]

[0070] In the formula, It is global average pooling. It is the ReLU activation function. This represents a 1×1 convolution. This represents a multilayer perceptron.

[0071] The spectral attention submodule then extracts the spectral information of the input features and generates empty spectral attention weights. The upper-level decoding feature is obtained by multiplying it with itself. as follows:

[0072]

[0073] In the formula, Represents a 7×7 convolution; spectral query tensor and spectral key tensor pass The results obtained after processing are as follows:

[0074]

[0075] In the formula, and It consists of 7×7 convolutions, batch normalized layers, and ReLU activation functions.

[0076] S4: After the hybrid spectral attention U-shaped deep neural network is trained, the hyperspectral image to be detected is input into the trained network after preprocessing in step S1 to obtain the saliency prediction map.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A salient target detection method based on a hybrid spectral attention U-shaped neural network, characterized in that, Includes the following steps: S1: Acquire a hyperspectral image and preprocess it to obtain a dimension-reduced spectral image, a spectral saliency image, and an RGB spectral mapping image; wherein, the dimension-reduced spectral image is used as the main mode input, and the spectral saliency image and the RGB spectral mapping image are stitched together to serve as the supplementary mode input; S2: Construct a hybrid spectral attention U-shaped deep neural network, which includes a two-stream encoder, a single-stream decoder, a bottleneck layer, and skip connections. The two-stream encoder is used to extract deep features of the main mode and the complementary mode, respectively. The bottleneck layer is located between the two-stream encoder and the single-stream decoder and is used to integrate the features of the two modes. The single-stream decoder is used to recover the feature space resolution layer by layer and generate a saliency prediction map. The skip connections are used to connect the features of corresponding levels in the two-stream encoder and the single-stream decoder. S3: Input the reduced spectral image and the supplementary modality input image into the hybrid spectral attention U-shaped deep neural network for training. Use the spectral saliency mixture loss function to intensively supervise the network and optimize the network parameters until the network converges. S4: The hyperspectral image to be detected is preprocessed in step S1 and then input into the trained neural network to obtain a saliency prediction map.

2. The salient target detection method based on a hybrid spectral attention U-shaped neural network according to claim 1, characterized in that, The dual-stream encoder includes a first branch and a second branch with identical structures. Each branch is divided into three stages. Each stage contains two cascaded dual-layer routing blocks and an image block merging layer composed of 3×3 convolutions. The first branch is used to encode the dimension-reduced spectral image, and the second branch is used to encode the supplementary modal image. The encoded features of the two branches are integrated at the end of the encoder by a multimodal spectral mixing frequency attention mechanism and then input into the bottleneck layer.

3. The salient target detection method based on a hybrid spectral attention U-shaped neural network according to claim 1, characterized in that, The single-stream decoder is divided into three stages. Each stage contains an image patch expansion layer consisting of 3×3 convolutions and two cascaded two-layer routing blocks. Each stage of the decoder generates a saliency prediction map and calculates the loss. The saliency prediction map and the decoded features are fed into the previous stage for optimization through skip connections.

4. A salient target detection method based on a hybrid spectral attention U-shaped neural network according to claim 2 or 3, characterized in that, The dual-layer routing block consists of a 3×3 depthwise convolution, two normalization layers, a dual-layer routing self-attention layer, three residual connections, and a dual-layer multilayer perceptron. The dual-layer routing self-attention layer is implemented based on a dynamic sparse dual-layer routing self-attention mechanism, which includes region partitioning and input projection, region-to-region routing, and token-to-token attention.

5. The salient target detection method based on a hybrid spectral attention U-shaped neural network according to claim 4, characterized in that, The region-to-region routing constructs a sparse routing index matrix by calculating the adjacency matrix between the query region and the key regions, and using the row-by-row top-k operator to filter out the k most relevant key regions for each query region.

6. The salient target detection method based on a hybrid spectral attention U-shaped neural network according to claim 2, characterized in that, The multimodal spectral mixing frequency attention mechanism includes a high-frequency attention head and a low-frequency attention head; the high-frequency attention head realizes the cross-interaction between the main modality coding features and the supplementary modality high-frequency edge features through a neighborhood attention mechanism; The low-frequency attention head uses a neighborhood attention mechanism to achieve self-attention interaction within the supplementary modality to extract low-frequency salient features; the output features of the high-frequency head and the low-frequency head are integrated to obtain hybrid frequency encoded features.

7. The salient target detection method based on a hybrid spectral attention U-shaped neural network according to claim 2, characterized in that, The space-spectral driven feature fusion skip connection includes a spatial attention submodule and a spectral attention submodule; The spatial attention submodule is used to generate spatial attention weights to compensate for the spatial information loss caused by encoder downsampling; The spectral attention submodule is used to generate spectral attention weights to enhance the utilization of spectral features.

8. The salient target detection method based on a hybrid spectral attention U-shaped neural network according to claim 7, characterized in that, The spatial attention submodule generates a spatial query tensor through global average pooling and 1×1 convolution, and generates a spatial key tensor through multilayer perceptron processing, and then calculates the spatial attention weights. The spectral attention submodule generates spectral query tensors and spectral key tensors through 7×7 convolution and batch normalization, and then calculates the spectral attention weights.

9. The salient target detection method based on a hybrid spectral attention U-shaped neural network according to claim 1, characterized in that, The spectral significance mixed loss function consists of three parts: binary cross-entropy loss, structural similarity index loss, and cross-union ratio loss, and is used to simultaneously supervise the final significance prediction map and intermediate significance prediction maps at all levels.

10. The salient target detection method based on a hybrid spectral attention U-shaped neural network according to claim 1, characterized in that, The dimensionality-reduced spectral image is obtained by uniformly sampling the original hyperspectral image along the channel dimension, with 50 channels; the spectral saliency image is obtained by calculating the center-periphery spectral contrast of the hyperspectral image; and the RGB spectral mapping image is obtained by converting the RGB spatial dimension of the hyperspectral image to the CIE XYZ spatial dimension.

Images