Salient target detection method of dual-process CNN-Transform network based on brain inspiration

Abstract

The invention relates to a brain inspiration-based dual-process CNN-Transform network saliency target detection method, which comprises the following steps of: constructing a data set for saliency target detection, and constructing a network model based on pyramid attention U-Net and RECA-Transform, the network model comprises a double-branch pyramid U-Net module and a residual efficient channel attention converter module; wherein the dual-branch pyramid U-Net module is an encoder-decoder architecture which comprises an encoder part and a decoder part; using a basic loss function and a boundary attention loss function to form an attention boundary optimization loss function to train the network model; and carrying out saliency target detection on the image by using the trained network model. The objective of the invention is to solve the problem of insufficient accuracy of the saliency target boundary and realize comprehensive perception and prediction of the saliency target boundary.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing technology in computer science, and in particular to a salient object detection method based on a brain-inspired dual-process CNN-Transformer network. Background Art

[0002] Salient object detection imitates the human visual system to locate and segment objects in visual scenes to quickly find the most important objects or regions. It has been widely used in various tasks in computer vision, such as object detection, semantic segmentation, AR / VR, graphic captioning, etc.

[0003] Among the existing salient object detection methods, most of them are still based on the multi-scale feature extraction architecture of convolutional neural network (CNN). These methods capture the global structure and local details of the image through a multi-scale feature fusion network, thereby improving the accuracy of salient prediction. However, the feature information obtained in this way is often redundant, which makes it difficult to accurately segment the most salient objects or regions in subsequent salient object detection. In addition, existing methods usually focus on the spatial features of the image and ignore the importance of visual attribute information in the channel dimension. In deep networks, channel-level features imply different visual attributes, which can provide a new information reference dimension to better support the salient object detection task. Recently, the Transformer-based encoder-decoder architecture has attracted widespread attention as an alternative to traditional methods. This type of architecture generates an attention-driven global representation through queries to explain the relationship between objects and their surrounding image context.

[0004] More and more studies have shown that the hierarchical operation of the human visual perception system is similar to the structure of the ventral visual stream (VVS), which inspired the design of the hybrid CNN-Transformer architecture. More and more evidence shows that brain recognition is based on the Dual-Process Detection Theory, which has had a profound impact on psychology and cognitive neuroscience. As an important framework, the dual-process detection theory aims to explain the cognitive mechanisms in human recognition and memory. In human visual saliency detection, when an object enters the visual scene, the brain immediately initiates a series of complex processing steps. This process can be explained in detail by the dual-process detection theory. Specifically, Fig.10As shown, the brain's visual processing is divided into two stages: familiarity and recollection. When the primary visual cortex receives light information from an object, it quickly and unconsciously begins to process basic features of the image, such as edges, directions, and basic shapes. This initial processing is similar to the function of convolutional neural networks, which quickly extract and identify local features of the image through a hierarchical structure. The brain then forms a preliminary sense of recognition of the object, a process called familiarity. If more detailed information is needed, the brain will engage in a slower and more conscious recollection process that involves the hippocampus and related memory areas. This stage requires integrating information from different sources to confirm the identity of the object and its importance in a specific context. This more complex process is similar to how Transformers work, capturing long-range dependencies and processing information comprehensively through self-attention mechanisms and global feature modeling.

[0005] Salient object detection not only requires identifying salient objects in an image, but also ensures that the predicted objects have precise and clear boundaries. Although relying solely on CNN's multi-scale feature fusion can improve feature extraction to a certain extent, it often leads to the loss of fine features and fails to retain complete information. On the other hand, the Transformer structure can efficiently process and model the global information and multi-scale features of an image, but it is still insufficient in capturing fine edge features. Therefore, the method of combining the Transformer structure with CNN's multi-scale feature fusion can better balance global information capture and boundary refinement, thereby achieving better results in salient object detection. Summary of the invention

[0006] The purpose of the present invention is to provide a salient object detection method based on a brain-inspired dual-process CNN-Transformer network, so as to solve the problems that the existing CNN method has feature redundancy in the salient object detection task, is difficult to accurately segment salient objects or regions, ignores the visual attribute information of the channel dimension, and the existing Transformer structure can capture global information but is still insufficient in capturing fine edge features.

[0007] The technical solution adopted by the present invention is as follows:

[0008] A method for salient object detection based on a brain-inspired dual-process CNN-Transformer network, comprising the following steps:

[0009] S1: construct a dataset for salient object detection, and divide the dataset into a training set, a test set, and a validation set;

[0010] S2: constructing a network model based on pyramid attention U-Net and RECA-Transformer, wherein the network model based on pyramid attention U-Net and RECA-Transformer includes a dual-branch pyramid U-Net module and a residual efficient channel attention converter module (RECA-Transformer module); wherein the dual-branch pyramid U-Net module is an encoder-decoder architecture including an encoder part and a decoder part;

[0011] The encoder part of the dual-branch pyramid U-Net module includes a pre-trained VGG16 network and a spatial attention module (SS); the pre-trained VGG16 network is used to gradually extract feature information of the input image, and at the same time, the spatial attention module (SS) is combined to form a pyramid structure, so as to more effectively extract the spatial feature representation X and visual feature V of the input image, and enhance the spatial feature representation X in combination with the visual feature V;

[0012] The decoder part of the dual-branch pyramid U-Net module uses transposed upsampling (TransposedUpsampling), linear interpolation upsampling (Bilinear Upsampling), spatial context pooling module (Sup) and residual coordinate attention model (RCAM); the transposed upsampling, linear interpolation upsampling and spatial context pooling modules are used interchangeably to retain the spatial information Y of the feature map; the residual coordinate attention module is used to generate the spatial attention feature map Y s ;

[0013] The residual efficient channel attention transformer (RECA-Transformer) includes a RECA transformer block (RECATransformer Block), which performs feature extraction and captures global characteristics through multi-layer attention mechanism and weight learning;

[0014] S3: Use the training set to train the network model based on the pyramid attention U-Net and RECA-Transformer, use the validation set to adjust the model parameters, determine whether it is overfitting, use the combination of the basic loss function BaseLoss and the boundary attention loss function EdgeLoss as the model's attention boundary optimization loss function TotalLoss, and complete the training of the network model based on the pyramid attention U-Net and RECA-Transformer; use the test set to test the network performance of the network model based on the pyramid attention U-Net and RECA-Transformer;

[0015] S4: Use the trained pyramid attention U-Net and RECA-Transformer based network models for salient object detection.

[0016] Furthermore, the encoder part of the dual-branch pyramid U-Net module includes five stages, namely, the first stage consisting of the 2nd to 5th layers of the VGG16 network and the spatial attention module, the second stage consisting of the 5th to 10th layers of the VGG16 network and the spatial attention module, the third stage consisting of the 10th to 17th layers of the VGG16 network and the spatial attention module, the fourth stage consisting of the 17th to 24th layers of the VGG16 network and the spatial attention module, and the fifth stage consisting of the 24th to 31st layers of the VGG16 network and the spatial attention module; wherein the VGG16 network uses pre-trained convolutional layers and pooling layers to extract feature information of the image, and in each convolution stage, the size of the feature map is successively reduced to 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image; the spatial attention module consists of a convolutional layer and an effective channel attention module, the convolutional layer in the spatial attention module is used to extract the spatial feature representation X of the feature map, and the effective channel attention module is used to enhance the spatial feature representation.

[0017] Furthermore, the expression of the convolutional layer in the spatial attention module is as follows:

[0018] X = (Conv2D 3×3 (ReLU(Conv2D 1×1 (x c )))),

[0019] Among them, X represents the spatial feature representation of the feature map, Conv2D 3×3 (·) represents a two-dimensional convolution, the convolution kernel size is 3×3, ReLU (·) represents the ReLU activation function, Conv2D 1×1 (·) represents a two-dimensional convolution, the convolution kernel size is 1×1, x cRepresents the input feature map of the spatial attention module;

[0020] The effective channel attention module includes a global average pooling layer GAP, a one-dimensional convolutional layer and a Sigmoid activation function stacked in sequence, and the output of the Sigmoid activation function is multiplied by the input of the global average pooling layer along the channel as the output of the effective channel attention module; the expression of the effective channel attention module is as follows:

[0021] X′=ECA(X)=X·σ(Conv1D 1×1 (AvgPool(X))),

[0022] Among them, X′ represents the features after enhancement processing, X represents the spatial feature representation of the feature map, ECA(·) represents the processing function of the effective channel attention module, σ(·) refers to the Sigmoid activation of the result of one-dimensional convolution to obtain the channel attention weight, Conv1D 1×1 (·) represents one-dimensional convolution, the convolution kernel size is 1×1, and AvgPool(·) represents the global average pooling operation;

[0023] That is, the expression of the spatial attention module is as follows:

[0024] X′=ECA(Conv2D 3×3 (ReLU(Conv2D 1×1 (x c )))).

[0025] Furthermore, the decoder part of the dual-branch pyramid U-Net module includes six stages, the first stage of the decoder part includes a residual coordinate attention module (RCAM) and a pixel reorganization module (PixelShuffle); the second stage of the decoder part includes a linear interpolation upsampling (Bilinear Upsampling), a spatial context pooling module (Sup), a residual coordinate attention module (RCAM), and slicing and concatenation operations (Slicing and Concatenation); the third, fourth, and fifth stages of the decoder part all include transposed upsampling (Transposed Upsampling), a linear interpolation upsampling (Bilinear Upsampling), a spatial context pooling module (Sup), a residual coordinate attention module (RCAM), and slicing and concatenation operations (Slicing and Concatenation); the sixth stage of the decoder part includes transposed upsampling (Transposed Upsampling), slicing and concatenation operations (Slicing and Concatenation).

[0026] Furthermore, the spatial context pooling module (Sup) first uses a 1×1 convolution operation to map the input feature map x of the spatial context pooling module to a single-channel feature map G, and the dimension of the obtained feature map G is (N, 1, H, W). Subsequently, the H channel and the W channel of the feature map G are combined together, and the H and W dimensions of the feature map G are merged to obtain a feature map G′, and the dimension of the feature map G′ is (N, 1, H×W); then a Softmax operation is performed to calculate the weight of each position of the feature map G′, and the feature map G′ after the channel merger is normalized and applied to the input feature map x, and the enhanced feature map is obtained by element-wise multiplication.

[0027] Furthermore, the residual coordinate attention module (RCAM) includes parallel horizontal global average pooling layers (XAvg Pool) and vertical global average pooling layers (Y Avg Pool), a concatenation operation (Conact), a 3×3 convolution layer, a batch normalization layer (Batch Norm), a non-linear activation layer (Non-linear), followed by two parallel 1×1 convolution layers and two parallel Sigmoid activation functions, the outputs of the two Sigmoid activation functions are expanded to the size of the input feature map, and then the expanded feature map is combined with the input feature map of the residual coordinate attention module through element-by-element multiplication, and the obtained output is added to the input feature map of the residual coordinate attention module through residual connection to complete the channel expansion of the input feature map. The residual coordinate attention module enhances the representation ability of the feature map by combining spatial and channel attention mechanisms; its principle is to generate the attention weight of each channel through global average pooling, 1×1 convolution and Sigmoid activation, and apply it to the input feature map, so that the model can better capture local and global feature information.

[0028] The residual coordinate attention module is used to perform channel expansion on the input feature map. First, the input feature map of the residual coordinate attention module is subjected to global average pooling in the horizontal and vertical directions respectively through the parallel horizontal global average pooling layer (X Avg Pool) and vertical global average pooling layer (Y Avg Pool) to extract the global average information and retain the spatial information; then, the horizontal and vertical pooled feature maps are concatenated in the second dimension to generate a new feature map for further processing and analysis; the specific workflow is as follows:

[0029] y = concat(x h ,x w , dim=2);

[0030] Among them, x h and x wThey represent the pooling results in the vertical and horizontal directions respectively, concat(·) represents the concatenation operation, and y represents the concatenated feature map;

[0031] Then, a 3×3 convolutional layer and a batch normalization layer are used to reduce the dimension of the concatenated feature map, and nonlinear characteristics are introduced through the ReLU activation function:

[0032] y′=ReLU(BN(Conv2D 3×3 (y)));

[0033] Among them, y′ represents the feature map after dimensionality reduction, ReLU(·) represents the ReLU activation function, BN(·) represents normalization, Conv2D 3×3 (·) represents a two-dimensional convolution with a kernel size of 3×3;

[0034] Next, the reduced feature map y′ is separated into horizontal feature maps x w1 and vertical feature map x h1 , and generate attention weights through a 1×1 convolutional layer and a Sigmoid activation function respectively:

[0035] x h1 ,x w1 =split(y′,[h,w],dim=2);

[0036] x h2 =Sigmoid(Conv2D 1×1 (x h1 ));

[0037] x w2 =Sigmoid(Conv2D 1×1 (x w1 ));

[0038] Among them, split(·) means splitting the feature map in the second dimension, Sigmoid(·) means the sigmoid activation function, x h2 represents the attention weight vertical feature map, x w2 Represents the attention weight level feature map;

[0039] Then, the generated attention weight vertical feature map x h2 and the attention weight level feature map x w2 Expand to the size of the input feature map and combine them with the input feature map by element-wise multiplication:

[0040] x h3 =x h2 expand(h,w);

[0041] x w3 =x w2 expand(h,w);

[0042] f = identity x h3 ·x w3 ;

[0043] Where f represents the output result after element-by-element multiplication, expand(h,w) represents the length and width of the attention weight feature map expanded to h and w, identity represents the initial input feature map of the residual coordinate attention module (RCAM), and x h3 represents the expanded vertical feature map, x w3 represents the expanded horizontal feature map; finally, the input and output features are combined through the residual connection.

[0044] Furthermore, the residual efficient channel attention converter module (RECA-Transformer module) first performs preliminary processing on the input data through a feature block partitioning module (Patch Partition), and the input data is divided into small blocks (patch), and each small block of input data is converted into a feature vector representation through a linear embedding module (Linear Embedding); the converted feature vector representation is sequentially processed by multiple consecutive RECA transformation blocks (RECATransformer Block×2) and linear embedding modules; then, the spatial resolution of the feature map is reduced through a feature block merging module (Patch Merging), and feature extraction is performed through two single RECA transformation blocks (RECATransformer Block×1) at the top of the model; the extracted data features are adjusted in resolution through a block expansion module (Patch Expanding), and the spatial resolution of the feature map is gradually increased to achieve upsampling, thereby generating high-resolution output features; and again through multiple consecutive RECA transformation blocks (RECATransformer The continuous RECA conversion block at this stage receives the feature information transmitted across layers by the continuous RECA conversion block at the front end; the extracted data features are processed step by step again and then the spatial resolution of the feature map is gradually reduced through the feature block merging module (Patch Merging), while the channel dimension is increased to achieve efficient feature extraction and calculation optimization; finally, the channel dimension of the feature map is adjusted using linear projection (LinearProjection) as output to obtain the final feature map.

[0045] Among them, the RECA Transformer Block introduces the effective channel attention module (ECA) and the residual MLP module (Re-MLP module), which aims to enhance useful features and suppress useless features through adaptive weighting of the channel dimension. Specifically, the effective channel attention module (ECA) calculates the attention weight of each channel and weights the channel features according to these weights, so that the model can focus more on key features, thereby more accurately capturing and utilizing important information in complex visual tasks.

[0046] Furthermore, a continuous RECA Transformer Block (RECA Transformer Block×2) includes two connected RECA Transformer Blocks (RECA Transformer Block), wherein the RECA Transformer Block (RECA Transformer Block) includes a layer normalization (i.e., LN), a window-based multi-head self-attention mechanism (i.e., W-MSA), a layer normalization (i.e., LN), a residual MLP module (Re-MLP module) and an effective channel attention module (ECA) connected in sequence, wherein the input of the first layer normalization (LN) is added to the output of the window-based multi-head self-attention mechanism (W-MSA) as the input of the second layer normalization (LN), and the input of the second layer normalization (LN) is added to the output of the effective channel attention module (ECA) as the output of the RECA Transformer Block; in the continuous RECA Transformer Block (RECA Transformer Block×2), the output of the first RECA Transformer Block (RECA Transformer Block) is used as the second RECA Transformer Block (RECA The input of the Transformer Block.

[0047] In addition, a residual connection is introduced in the residual MLP module (Re-MLP module), which includes an effective channel attention module (ECA), a fully connected layer fc1, a fully connected layer fc2, and a fully connected layer fc3 connected in sequence, wherein the residual connection acts on the fully connected layer fc1 and the fully connected layer fc2, and the output of the fully connected layer fc1 is added to the output of the fully connected layer fc2 as the input of the fully connected layer fc3; the residual connection not only strengthens the role of the two fully connected layers fc1 and fc2, but also effectively alleviates the gradient vanishing problem; by introducing the residual connection, the gradient in the model can flow smoothly during the back propagation process, which promotes the effective propagation of the gradient.

[0048] Furthermore, the calculation formula of the attention boundary optimization loss function TotalLoss composed of the basic loss function BaseLoss and the boundary attention loss function EdgeLoss is as follows:

[0049] TotalLoss=BaseLoss+β·EdgeLoss;

[0050]

[0051] AttentionWeights=1+α·Edges;

[0052]

[0053] Among them, BaseLoss is the basic loss, which calculates the cross entropy loss between the output image predicted by the model and the real label (targets). The edge map (edges) is generated by calculating the gradient components of the real mask map in the vertical and horizontal directions to obtain a high-quality edge map. β is the weight of the edge loss, EdgeLoss is the edge optimization loss, and CrossEntropy (p i ,t i ) represents the cross entropy loss function, p i represents the predicted value, t i Represents the actual value, N represents a total of N points, AttentionWeights represents the attention weight, α represents the proportion of edge weight, and Edges represents the edge.

[0054] The beneficial effects of the present invention are:

[0055] (1) Based on the dual process detection theory, the present invention applies CNN and Transformer to salient target detection, forming a hybrid architecture based on the pyramid attention U-Net and residual efficient channel attention transformer model. The dual-branch pyramid attention U-Net module (CNN module) is responsible for extracting broad and general shallow features, while the residual efficient channel attention transformer module (Transformer module) generates high-level representations of specific tasks and captures global relationships through the attention mechanism. By combining the advantages of both, the present invention significantly improves model performance and prediction accuracy.

[0056] (2) The present invention combines visual features through the spatial attention module and the residual coordinate attention module to enhance the spatial feature representation and balance the model's attention to regions of different sizes. In addition, the present invention constructs local region features that incorporate visual attribute information in the channel dimension, which helps the model retain more local information of the image context.

[0057] (3) The present invention constructs a residual efficient channel attention transformer module (RECA-Transformer module) and an attention boundary optimization loss function, which are used to capture the global context information of the image and perform the image boundary prediction task respectively. The method of the present invention can effectively solve the problem in the prior art that it is difficult to achieve the integrity of fine edge features when predicting salient targets, and generate more accurate salient targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 It is a structural diagram of a network model based on pyramid attention U-Net and RECA-Transformer in an embodiment of the present invention;

[0060] Figure 2 Schematic diagram of the structure of the residual coordinate attention module in an embodiment of the present invention;

[0061] Figure 3 Schematic diagram of the structure of the effective channel attention module (ECA) in an embodiment of the present invention;

[0062] Figure 4 Schematic diagram of the structure of the Re-MLP module in an embodiment of the present invention;

[0063] Figure 5 Schematic diagram of the structure of a continuous RECA transformation block (RECA-Transformer Block×2) in an embodiment of the present invention;

[0064] Figure 6 It is a Precision-Recall evaluation index diagram in an embodiment of the present invention;

[0065] Figure 7 The figure is a comparison visualization result of the method of the present invention and other ten existing salient object detection methods;

[0066] Figure 8 This is a detection result diagram of module ablation visualization performed by the method of the present invention;

[0067] Fig. 9 This is a simplified flowchart of the present invention for building a network framework based on brain inspiration.

[0068] Fig.10 Schematic diagram of the architecture of the dual-process ventral visual stream (VVS) identified for the brain. DETAILED DESCRIPTION

[0069] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and therefore, the present invention is not limited to the limitations of the specific embodiments disclosed below.

[0070] Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with ordinary skills in the field described in this application. The "first", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one" or "one" do not indicate quantity restrictions, but indicate the existence of at least one. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0071] like Figure 1 As shown in FIG. 1 , a salient object detection method based on a brain-inspired dual-process CNN-Transformer network comprises the following steps:

[0072] Step S1: construct a data set for salient object detection, and divide the data set into a training set, a test set, and a validation set. In the embodiment of the present invention, five data sets widely used in the field of salient object detection are used, namely, the ECSSD data set, the PASCAL-S data set, the HKUIS data set, the DUT-OMRON data set, and the DUTS data set, and the data in the five data sets are divided into a training set, a test set, and a validation set according to a ratio of 7:2:1.

[0073] The ECSSD dataset contains 1,000 high-resolution images with complex backgrounds and diverse scenes. The salient objects in each image are of various types and have prominent locations. Compared with other datasets, the ECSSD dataset has precise annotations and is suitable for evaluating the performance of salient object detection algorithms when processing complex scenes.

[0074] The PASCAL-S dataset contains 850 images with complex scenes, diverse backgrounds, and multiple categories of salient objects. Each image has been carefully annotated to ensure accurate representation of the object. Compared with the ECSSD dataset, the PASCAL-S dataset has slightly fewer images, but it places higher requirements on the adaptability of the algorithm under different backgrounds.

[0075] The HKUIS dataset contains 4447 images and is challenging because it contains multiple unconnected salient objects, overlapping boundaries, and low color contrast. These characteristics make it particularly valuable in testing the robustness of salient object detection models. Compared with the ECSSD dataset and the PASCAL-S dataset, the HKUIS dataset is larger in number and focuses more on detection effects in difficult scenarios.

[0076] The DUT-OMRON dataset contains 5168 images, each of which shows one or two salient objects with rich details and rich and diverse visual features. Compared with the HKUIS dataset, the DUT-OMRON dataset has higher object richness and diversity, and is suitable for comprehensively evaluating the performance of detection methods under complex object conditions.

[0077] The DUTS dataset contains 15,572 images and is the largest public dataset in the field of salient object detection. The dataset is divided into a training set (DUTS-TR, 10,553 images) and a test set (DUTS-TE, 5,019 images), covering a variety of scenes and object categories. Compared with the ECSSD dataset, the PASCAL-S dataset, the HKUIS dataset, and the DUT-OMRON dataset, the DUTS dataset has obvious advantages in data scale and scene diversity, and can comprehensively test the generalization ability of the model.

[0078] Step S2: Construct a network model based on pyramid attention U-Net and RECA-Transformer, wherein the network model based on pyramid attention U-Net and RECA-Transformer includes a dual-branch pyramid U-Net module and a residual efficient channel attention converter module (RECA-Transformer module), wherein the dual-branch pyramid U-Net module is an encoder-decoder architecture including an encoder part and a decoder part.

[0079] The encoder part of the dual-branch pyramid U-Net module uses the pre-trained VGG16 network as the backbone network and combines the spatial attention module (SS) to form a pyramid structure; the convolutional layer and pooling layer in the pre-trained VGG16 network are used to gradually extract the feature information of the input image. At the same time, the pre-trained VGG16 network is combined with the spatial attention module (SS) to form a pyramid structure, thereby more effectively extracting the visual features V of the input image. The spatial attention module (SS) consists of a convolutional layer and an effective channel attention module (ECA), which is used to enhance the spatial feature representation X in combination with the visual features V.

[0080] The decoder part of the dual-branch pyramid U-Net module includes six stages, which use transposed upsampling, bilinear upsampling, spatial context pooling module (Sup) and residual coordinate attention model (RCAM); the transposed upsampling, bilinear upsampling and spatial context pooling modules are used interchangeably to retain the spatial information y of the feature map; the residual coordinate attention module is used to generate the spatial attention feature map Y s .

[0081] The residual efficient channel attention transformer module (RECA-Transformer module) introduces an effective channel attention module (Efficient Channel Attention, i.e. ECA) and a residual MLP module (Residual-MLP, i.e. Re-MLP module) on the basis of the Swin-Transformer backbone network.

[0082] The encoder part in the dual-branch pyramid U-Net module used in the embodiment of the present invention uses a pre-trained VGG16 network as the backbone network, which is divided into five stages, namely, the first stage consisting of the 2nd to 5th layers of the VGG16 network and a spatial attention module, the second stage consisting of the 5th to 10th layers of the VGG16 network and a spatial attention module, the third stage consisting of the 10th to 17th layers of the VGG16 network and a spatial attention module, the fourth stage consisting of the 17th to 24th layers of the VGG16 network and a spatial attention module, and the fifth stage consisting of the 24th to the last layer (i.e., the 31st layer) of the VGG16 network and a spatial attention module. The VGG16 network uses pre-trained convolutional layers and pooling layers to gradually extract the feature information of the image. In each convolution stage, the size of the feature map is reduced in sequence, starting from the original image and gradually reduced to 1 / 2 (stage 1), 1 / 4 (stage 2), 1 / 8 (stage 3), 1 / 16 (stage 4) and 1 / 32 (stage 5), forming a pyramid structure. like Figure 1 As shown in FIG. 1 , the input feature map first passes through the 2nd to 5th layers of the VGG16 network in the first stage of the encoder part and the spatial attention module to extract features respectively, and the extracted feature information is fused to obtain feature information g1, which is then input to the second stage of the encoder part for processing; the 5th to 10th layers of the VGG16 network in the second stage of the encoder part and the spatial attention module extract features from the feature information g1 respectively, and the extracted feature information is fused to obtain feature information g2, which is then input to the third stage of the encoder part for processing; the 10th to 17th layers of the VGG16 network in the third stage of the encoder part and the spatial attention module extract features from the feature information g2 respectively, and the extracted feature information is fused to obtain feature information g3. The feature information g3 is input into the fourth stage of the encoder for processing; the 17th to 24th layers of the VGG16 network and the spatial attention module in the fourth stage of the encoder respectively extract features from the feature information g3, and fuse the extracted feature information to obtain feature information g4, which is then input into the fifth stage of the encoder for processing; the 24th to the last layer (i.e., the 31st layer) of the VGG16 network in the fifth stage of the encoder and the spatial attention module respectively extract features from the feature information g4, and fuse the extracted feature information to obtain feature information g5; that is, the five stages of the encoder sequentially output feature information g1, feature information g2, feature information g3, feature information g4, and feature information g5 with gradually decreasing sizes.

[0083] The spatial attention module (SS) consists of a convolutional layer and an effective channel attention module (ECA). The convolutional layer of the spatial attention module is used to extract the spatial feature representation X of the feature map, and the effective channel attention module is intended to enhance the spatial feature representation X. Figure 3 As shown, the effective channel attention module (ECA) includes a global average pooling layer GAP, a one-dimensional convolutional layer and a Sigmoid activation function stacked in sequence, and the output of the Sigmoid activation function is multiplied by the input of the global average pooling layer along the channel as the output of the effective channel attention module;

[0084] The expression of the convolutional layer of the spatial attention module is as follows:

[0085] X = (Conv2D 3×3 (ReLU(Conv2D 1×1 (x c )))),

[0086] Among them, X represents the spatial feature representation of the feature map, Conv2D 3×3 (·) represents a two-dimensional convolution, the convolution kernel size is 3×3, ReLU (·) represents the ReLU activation function, Conv2D 1×1 (·) represents a two-dimensional convolution, the convolution kernel size is 1×1, x c Represents the input feature map of the spatial attention module;

[0087] The expression of the effective channel attention module is as follows:

[0088] X′=ECA(X)=X·σ(Conv1D 1×1 (AvgPool(X))),

[0089] Among them, X′ represents the features after enhancement processing, X represents the spatial feature representation of the feature map, ECA(·) represents the processing function of the effective channel attention module, σ(·) refers to the Sigmoid activation of the result of one-dimensional convolution to obtain the channel attention weight, Conv1D 1×1 (·) represents a one-dimensional convolution with a kernel size of 1×1, and AvgPool(·) represents a global average pooling operation.

[0090] That is, the expression of the spatial attention module of the present invention is as follows:

[0091] X′=ECA(Conv2D 3×3 (ReLU(Conv2D 1×1 (x c )))).

[0092] In the embodiment of the present invention, the decoder part of the dual-branch pyramid U-Net module includes six stages, such as Figure 1 As shown in Figure 1, the structure of the decoder part is described in detail as follows. In the first stage of the decoder part, in view of the lack of corresponding high-level features, the feature information g5 output by the fifth stage (Stage 5) of the encoder part is first processed. The feature information g5 obtained from the fifth stage of the encoder part of the dual-branch pyramid U-Net module is input to the first stage of the decoder part. The structure of the first stage of the decoder part is as follows: Figure 1 As shown in (a), the first stage of the decoder is processed by the Residual Coordinate Attention Model (RCAM) and the Pixel Shuffle module to obtain feature information S5. The basic principle of the Residual Coordinate Attention Module (RCAM) is to improve the representation ability of the feature map by coordinating the spatial and channel attention mechanisms; the Residual Coordinate Attention Module uses global average pooling, 1×1 convolution and Sigmoid activation function to generate attention weights for each channel and apply them to the input feature map, thereby effectively capturing local and global features.

[0093] The obtained feature information S5 is input to the second stage of the decoder for processing. The structure of the second stage of the decoder is as follows: Figure 1 As shown in (b), the second stage of the decoder first processes the feature information S5 through the linear interpolation upsampling (BilinearUpsampling) and the spatial context pooling module (Sup), and then sums it with the feature information g4 output by the fourth stage (Stage4) of the encoder to obtain the feature information R 21 , and then the feature information R 21 Input to the residual coordinate attention module (RCAM) for processing to obtain feature information R 22 , and then the feature information S5 and feature information R input in the first stage of the decoder 22 Slicing and concatenation are performed to obtain feature information R 23 , and finally the feature information R 23 The feature information S5 is summed up and the feature information S4 after the second stage processing of the decoder part is output.

[0094] The feature information S4 enters the third stage of the decoder for processing. The structure of the third stage of the decoder is as follows: Figure 1 As shown in (c), the feature information S4 is first transposed upsampling to obtain the feature information R 31At the same time, the feature information g3 transmitted from the third stage (Stage3) of the encoder is processed by linear interpolation upsampling (Bilinear Upsampling) and spatial context pooling module (Sup) to obtain feature information R 32 , and then the feature information R 32 Sum it with the feature information g4 from the fourth stage (Stage 4) of the encoder to obtain the fused feature information R 33 , and then use the residual coordinate attention module (RCAM) to fused feature information R 33 Perform spatial enhancement to obtain feature information R 34 , and then the feature information R 34 With feature information R 31 Slicing and concatenation are performed together to obtain feature information R 35 This processing method enables the model to fully consider high-level and low-level feature information of different scales, thereby ensuring that the final generated feature map has high resolution and high quality. The final output feature information R 35 Again, and feature information R 31 The sum is performed and the feature information S3 after the third stage processing of the decoder part is output.

[0095] The feature information S3 enters the fourth stage of the decoder for processing. The structure of the fourth stage of the decoder is as follows: Figure 1 As shown in (c), the processing process is the same as the third stage of the decoder. First, the feature information S3 is transposed upsampling to obtain the feature information R 41 At the same time, the feature information g2 transmitted from the second stage (Stage2) of the encoder is processed by linear interpolation upsampling (Bilinear Upsampling) and spatial context pooling module (Sup) to obtain the feature information R 42 , and then the feature information R 42 Sum it with the feature information g3 from the third stage (Stage 3) of the encoder to obtain the fused feature information R 43 , and then the residual coordinate attention module (RCAM) is used for spatial enhancement to obtain feature information R 44 , and then the feature information R 44 With feature information R 41 Slicing and concatenation are performed to obtain feature information R 45 , the final output feature information R 45 Again, and feature information R 41The sum is performed and the feature information S2 after the fourth stage processing of the decoder part is output.

[0096] The feature information S2 enters the fifth stage of the decoder. The structure of the fifth stage of the decoder is as follows: Figure 1 As shown in (c), similarly, the feature information S2 is first transposed upsampling to obtain the feature information R 51 At the same time, the feature information g1 transmitted from the first stage (Stage1) of the encoder is processed by linear interpolation upsampling (BilinearUpsampling) and spatial context pooling module (Sup) to obtain feature information R 52 , and then the feature information R 52 Sum the feature information g2 from the second stage (Stage 2) of the encoder to obtain the feature information R 53 , and then the residual coordinate attention module (RCAM) is used for spatial enhancement to obtain feature information R 54 , and then the feature information R 54 With feature information R 51 Slicing and concatenation are performed to obtain feature information R 55 , the final output feature information R 55 Again, and feature information R 51 The sum is performed and the feature information S1 after the fifth stage of processing is output.

[0097] The obtained feature information S1 is then input into the sixth stage of the decoder for processing, such as Figure 1 As shown in (d) in the sixth stage of the decoder, the feature information S1 is first transposed upsampling to obtain the feature information R 61 , and then the feature information R 61 Slicing and concatenation are performed with the feature information g1 transmitted from the first stage (Stage 1) of the encoder part to obtain the feature information processed by the encoder-decoder architecture of the dual-branch pyramid U-Net module.

[0098] The working principle of the spatial context pooling module (Sup) is as follows: First, a 1×1 convolution operation is applied to the input feature map x of the spatial context pooling module, which is mapped to a single-channel feature map G, and the dimension of the output feature map G is (N, 1, H, W). Subsequently, the H channel and the W channel are combined together, and the H and W dimensions of the feature map G are merged, that is, (N, 1, H×W), to obtain the feature map G′. Then a Softmax operation is performed to calculate the weight of each position of the feature map G′, as described below:

[0099] G′=C reshape = reshape(G,(N,1,H×W));

[0100] A=Softmax(C reshape , dim=2);

[0101] Among them, reshape(·) means adjusting the size of the feature map, C reshape , dim = 2 means processing on the second dimension, Softmax(·) means Softmax operation, and A means the feature map obtained after normalization by Softmax operation.

[0102] The purpose of the Softmax operation is to normalize the weights so that the sum of the weights of all positions on the feature map is 1, and the weight corresponding to each position represents the importance of the corresponding position. Specifically, the Softmax operation normalizes the feature map G′ after channel merging and applies it to the original input feature map x (i.e., the input feature map of the spatial context pooling module), and obtains the enhanced feature map by element-wise multiplication. This process can be summarized as follows:

[0103] A=Softmax(C reshape ,dim=2)=torch.matmul(x.unsqueeze(1),G);

[0104] In the above formula, A represents the feature map obtained after the Softmax operation normalization, torch.matmul(·) represents matrix multiplication, unsqueeze(1) represents the removal of the first dimension, and x represents the input feature map of the spatial context pooling module.

[0105] The residual coordinate attention module (RCAM) in the embodiment of the present invention uses a recurrent convolutional attention mechanism. The residual coordinate attention module (RCAM) can effectively model long-range dependencies in feature maps, improve the contextual understanding ability of the model, and enhance the confidence of the predicted image. By capturing these dependencies, the residual coordinate attention module (RCAM) not only maintains the fine boundary structure, but also strengthens the network's ability to aggregate and interpret the spatial and contextual information of each region of the input image, thereby achieving more robust and reliable predictions. Figure 2 As shown in the figure, the residual coordinate attention module includes parallel horizontal global average pooling layers (X Avg Pool) and vertical global average pooling layers (Y Avg Pool), a concatenation operation (Conact), a 3×3 convolution layer, a batch normalization layer (Batch Normalization, i.e., Batch Norm), and a nonlinear activation layer (Non-linear), followed by two parallel 1×1 convolution layers and two parallel Sigmoid activation functions. The outputs of the two Sigmoid activation functions are expanded to the size of the input feature map, and then the expanded feature map is combined with the input feature map of the residual coordinate attention module through element-by-element multiplication, and the obtained output is added to the input feature map of the residual coordinate attention module through residual connection to complete the channel expansion of the input feature map. Using a 3×3 deep convolution layer to extract spatial features on each channel can effectively extract spatial features and maintain computational efficiency, thereby enhancing the spatial expression ability of the feature map.

[0106] Specifically, first, the input feature map of the residual coordinate attention module is subjected to global average pooling in the horizontal and vertical directions respectively through the parallel horizontal global average pooling layer (X Avg Pool) and vertical global average pooling layer (Y Avg Pool) to extract the global average information and retain the spatial information. Subsequently, the horizontal and vertical pooled feature maps are concatenated in the second dimension to generate a new feature map for further processing and analysis. The specific workflow is as follows:

[0107] y = concat(x h ,x w , dim=2);

[0108] Among them, x h Pooling result in vertical direction, x w represents the pooling result in the horizontal direction, concat(·) represents the concatenation operation, and y represents the concatenated feature map. Then, a 3×3 convolution layer and a batch normalization layer are used to reduce the dimension of the concatenated feature map, and a nonlinear characteristic is introduced through the ReLU activation function:

[0109] y′=ReLU(BN(Conv2D 3×3 (y)));

[0110] Among them, y′ represents the feature map after dimensionality reduction, ReLU(·) represents the ReLU activation function, BN(·) represents normalization, Conv2D 3×3 (·) represents a two-dimensional convolution with a kernel size of 3×3.

[0111] Next, the reduced feature map y′ is separated into horizontal feature maps x w1 and vertical feature map x h1 , and generate attention weights through a 1×1 convolutional layer and a Sigmoid activation function respectively:

[0112] x h1 ,x w1 =split(y′,[h,w],dim=2);

[0113] x h2 =Sigmoid(Conv2D 1×1 (x h1 ));

[0114] x w2 =Sigmoid(Conv2D 1×1 (x w1 ));

[0115] Among them, split(·) means splitting the feature map in the second dimension, Sigmoid(·) means the sigmoid activation function, x h2 represents the attention weight vertical feature map, x w2 Represents the attention weight level feature map.

[0116] Then, the generated attention weight vertical feature map x h2 and the attention weight level feature map x w2 Expand to the size of the input feature map and combine them with the input feature map by element-wise multiplication:

[0117] x h3 =x h2 expand(h,w);

[0118] x w3 =x w2 expand(h,w);

[0119] f = identity x h3 ·x w3 ;

[0120] Where f represents the output result after element-by-element multiplication, expand(h,w) represents the length and width of the attention weight feature map expanded to h and w, identity represents the initial input feature map of the residual coordinate attention module (RCAM), and x h3 represents the expanded vertical feature map, x w3 represents the expanded horizontal feature map; finally, the input and output features are combined through the residual connection.

[0121] The structure of the residual efficient channel attention transformer module (RECA-Transformer module) of the embodiment of the present invention is as follows: Figure 1 As shown in Figure 1. First, the input data is preliminarily processed by the feature block partition module (Patch Partition). The input data is divided into small patches (patches), and each small patch of input data is converted into a feature vector representation through a linear embedding module (Linear Embedding). The converted feature vector representation is processed step by step through multiple consecutive RECA transformation blocks (RECATransformer Block×2) and linear embedding modules. Then, the spatial resolution of the feature map is reduced by a specific feature block merging module (Patch Merging), and processed by two single RECA transformation blocks (RECATransformer Block×1) on the top of the model. Among them, the RECA transformation block (RECA Transformer Block) is used to perform feature extraction on the feature vector representation, and capture global characteristics through multi-layer attention mechanism and weight learning. Multiple consecutive RECA transformation blocks and linear embedding modules gradually optimize the features to ensure that important information can be gradually extracted through multi-level processing. The extracted data features are adjusted in resolution through a specific patch expansion module (Patch Expanding), gradually increasing the spatial resolution of the feature map to achieve upsampling, thereby generating high-resolution output features. It is processed again by multiple continuous RECA Transformer Blocks (RECA Transformer Block×2) and linear embedding modules. The continuous RECA transformation block at this stage will receive the feature information transmitted across layers by the continuous RECA transformation block at the front end, so that the feature information extracted in the early stage can be effectively accessed by subsequent modules, improving the overall modeling ability. The extracted data features are processed again step by step, and then the spatial resolution of the feature map is gradually reduced through a specific feature block merging module (Patch Merging), while the channel dimension is increased to achieve efficient feature extraction and calculation optimization. Finally, the channel dimension of the feature map is adjusted using linear projection (LinearProjection) as the output to obtain the final feature map.

[0122] Among them, the RECA Transformer Block includes an effective channel attention module (ECA) and a residual MLP module (Re-MLP module) to enhance useful features and suppress useless features through adaptive weighting in the channel dimension. Specifically, the effective channel attention module (ECA) calculates the attention weight of each channel and weights the channel features according to these weights, so that the model can focus more on important features, thereby more accurately capturing and utilizing key information when processing complex visual tasks. The RECATransformer Block×2 module in this embodiment represents a continuous RECA transformation block, which contains two connected RECA transformation blocks (RECATransformer Block); RECATransformerBlock×1 represents a single RECA transformation block (RECATransformer Block). As shown Figure 5 As shown, the RECA Transformer Block includes a layer normalization (i.e., LN), a window-based multi-head self-attention mechanism (i.e., W-MSA), a layer normalization (i.e., LN), a residual MLP module (Re-MLP module), and an effective channel attention module (ECA) connected in sequence, wherein the input of the first layer normalization (LN) is added to the output of the window-based multi-head self-attention mechanism (W-MSA) as the input of the second layer normalization (LN), and the input of the second layer normalization (LN) is added to the output of the effective channel attention module (ECA) as the output of the RECA Transformer Block; at the same time, in the continuous RECA Transformer Block (RECATransformer Block×2), the output of the first RECA Transformer Block (RECATransformer Block) is used as the input of the second RECA Transformer Block (RECATransformer Block). In addition, a residual connection is introduced in the residual MLP module (Re-MLP module), as shown in Figure 4 As shown in the figure, the residual MLP module includes an effective channel attention module (ECA) connected in sequence, three fully connected layers fc1, fc2, and fc3, where the residual connection acts on the fully connected layer fc1 and the fully connected layer fc2, and the output of the fully connected layer fc1 is added to the output of the fully connected layer fc2 as the input of the fully connected layer fc3, which not only enhances the role of the two fully connected layers fc1 and fc2, but also effectively alleviates the gradient vanishing problem, promotes the smooth flow of gradients in the back propagation process, and improves the training effect of the model.

[0123] Step S3: Use the training set to train the network model based on the pyramid attention U-Net and RECA-Transformer constructed in step S2, use the validation set to adjust the model parameters, determine whether it is overfitting, and use the combination of the basic loss function BaseLoss and the boundary attention loss function EdgeLoss as the model's attention boundary optimization loss function TotalLoss to complete the training of the network model based on the pyramid attention U-Net and RECA-Transformer; use the test set to test the network performance of the network model based on the pyramid attention U-Net and RECA-Transformer; the calculation formula of the attention boundary optimization loss function TotalLoss composed of the basic loss function BaseLoss and the boundary attention loss function EdgeLoss is as follows:

[0124] TotalLoss=BaseLoss+β·EdgeLoss;

[0125]

[0126] AttentionWeights=1+α·Edges;

[0127]

[0128] Among them, BaseLoss is the basic loss function, which calculates the cross entropy loss between the model's predicted output image and the true label (targets). The edge map (edges) is generated by calculating the gradient components of the true mask map in the vertical and horizontal directions, thereby obtaining a high-quality edge map. β is the weight of the edge loss, and EdgeLoss is the boundary attention loss function. CrossEntropy(p i ,t i ) represents the cross entropy loss function, p i represents the predicted value, t i Represents the actual value, N represents a total of N points, AttentionWeights represents the attention weight, α represents the proportion of edge weight, and Edges represents the edge.

[0129] Step S4: Use the trained network model based on pyramid attention U-Net and RECA-Transformer to perform salient object detection on the image to be detected.

[0130] The technical effects of the embodiments of the present invention are described below in conjunction with specific experimental data:

[0131] The embodiment of the present invention uses the PyTorch 2.3 framework during the training process, and the hardware configuration includes an i9-13900K CPU and an RTX 3090 GPU (24GB). The optimizer uses stochastic gradient descent (SGD), the initial learning rate is set to 0.005, and the momentum is 0.9. The batch size is 15. The training, test, and validation data are divided in a ratio of 7:2:1. During the training process, the image size is adjusted to 352×352. Note that the parameters of the boundary optimization loss function TotalLoss are set to β=2 and α=1.5.

[0132] In the experiment, five commonly used salient object detection evaluation indicators are used to evaluate the performance of the model, namely F-measure, Enhanced-alignment Measure (E-measure), Mean Absolute Error (MAE), Precision-Recall and Structure-measure (S-measure). The details of the evaluation indicators are described as follows:

[0133] (1) F-measure is a comprehensive indicator for evaluating model performance. It combines precision and recall to balance the trade-off between the two. The calculation formula is as follows:

[0134]

[0135] Among them, F β It represents the result value of F-measure, and β1 is a trade-off parameter, which is usually set to 1, indicating the balance between precision and recall. The higher the result value of F-measure, the better the overall performance of the model in detecting significant targets.

[0136] (2) Enhanced-alignment Measure (E-measure) aims to evaluate the structural similarity between the predicted saliency map and the true saliency map. It comprehensively considers the similarity at the pixel level and the overall structural information. The calculation formula is as follows:

[0137]

[0138] Among them, E φ represents the result value of the enhanced alignment metric, P φ and R φ They represent the adjusted precision and recall respectively. E-measure pays more attention to global consistency in saliency detection evaluation. φThe larger the value, the higher the structural similarity between the detection result and the true target.

[0139] (3) Mean Absolute Error (MAE) is used to measure the pixel difference between the predicted saliency map and the true saliency map. Its calculation formula is as follows:

[0140]

[0141] Among them, P(i,j) represents the pixel value of the predicted saliency map, G(i,j) represents the pixel value of the true saliency map, and H×W represents the total number of pixels in the image. The lower the MAE value, the closer the saliency map predicted by the model is to the true saliency map.

[0142] (4) The precision-recall curve is used to evaluate the performance of the model at different thresholds. Precision represents the proportion of pixels that are actually salient objects among the pixels predicted by the model as salient areas, and recall represents the proportion of correctly detected salient objects among all actual salient objects. By plotting the precision-recall curve at different thresholds, the performance of the model can be intuitively evaluated.

[0143] (5) Structural similarity measure (S-measure) is used to measure the structural similarity between the saliency map and the true saliency map. S-measure combines region-aware similarity and object-aware similarity, and the calculation formula is as follows:

[0144] S α =α1×S o +(1-α1)×S r ;

[0145] Among them, S α Represents the result value of the structural similarity measurement, S o Indicates object similarity, S r It represents the regional similarity, and α1 is the weight coefficient, which is usually set to 0.5. The larger the S-measure result value, the more consistent the structure of the saliency map predicted by the model is with the true saliency map.

[0146] To verify the effectiveness of the method, the salient object detection method of the present invention was tested on five datasets (i.e., ECSSD dataset, PASCAL-S dataset, HKU-IS dataset, DUT-OMRON dataset, and DUTS dataset), and compared with 10 neural network-based salient object detection methods published in recent years based on five evaluation indicators (i.e., F-measure, E-measure, MAE, Precision-Recall, and S-measure). The existing comparison methods include F3Net, LDF, MSFNet, PAKRN, SelfReformer, EDN, BBRF, MENet, CSD, and A2S-v3. The salient images of all comparison methods are provided by the authors of each method or generated by running their public codes. As shown in Table 1, in terms of the four evaluation indicators of F-measure, MAE, E-measure and S-measure (where the higher the F-measure and S-measure values, the better, the lower the MAE value, the better, and the higher the E-measure value, the better), the method of the present invention is generally better than all the comparison methods, especially on the ECSSD dataset, PASCAL-S dataset and DUT-OMRON dataset; on the HKU-IS dataset and DUTS dataset, the method of the present invention is not much different from the method with the best performance in other evaluation indicators. Precision-Recall is also a commonly used evaluation indicator in salient target detection, which is used to measure the detection accuracy and balance ability of the salient target detection model at different thresholds, and is particularly suitable for processing scenarios with unbalanced datasets. As Figure 6 As shown in the figure, the method of the present invention outperforms all the compared methods in terms of the Precision-Recall evaluation index.

[0147] Table 1 quantitatively evaluates our method with ten other techniques on five benchmark datasets (the bold data represent the best evaluation experimental data of all methods and the experimental evaluation data of our method in the last row):

[0148]

[0149] The experimental results in Table 1 show that our experimental evaluation results in the last row are generally significantly better than the previous methods, proving that our method has superior performance in salient target detection. For example, compared with the other ten methods, the best results were achieved in F-measure, MAE and S-measure on the ECSSD dataset, the best results were achieved in E-measure and S-measure on the PASCAL-S dataset, the best performance was achieved in F-measure, MAE and E-measure on the DUT-OMRON dataset, and the evaluation indicators on the HKU-IS and DUTS datasets were also high. In order to verify the performance of the method proposed in the present invention, as shown in Figure 1, Figure 7 As shown, the visualization superiority of the invented method is demonstrated.

[0150] In order to verify the effectiveness of the various modules proposed in the present invention, an ablation comparison experiment was carried out to analyze the results of four evaluation indicators, namely, the F-measure (F-measure), mean absolute error (MAE), enhanced alignment measure (E-measure), and structural similarity measure (S-measure), of a single dual-branch pyramid attention U-Net module, a network model based on pyramid attention U-Net and RECA-Transformer using a general boundary loss function module (cross entropy loss and Dice loss), a single residual efficient channel attention converter module, and a complete network model based on pyramid attention U-Net and RECA-Transformer of the present invention on the DUTS dataset, PASCAL-S dataset, and ECSSD dataset. The detection results are shown in Table 2.

[0151] Table 2:

[0152]

[0153] As can be seen from Table 2, ablation experiments were conducted on the dual-branch pyramid attention U-Net module (CNN module), the residual efficient channel attention transformer module (RECA-Transformer module) and the network model based on the pyramid attention U-Net and RECA-Transformer using the general boundary loss function module, which shows that these modules play an important role in the invented method and verifies the contribution of the invented method in improving the performance of the salient object detection task. Figure 8 The visualization superiority of the invented method is demonstrated.

[0154] Fig. 9This is a simplified flow chart of the network model built based on brain inspiration in the present invention. Specifically, the upper-layer network dual-branch pyramid attention U-Net module (CNN module) is used to extract extensive and common shallow features, while the lower-layer network residual efficient channel attention transformer module (RECA-Transformer module) focuses on generating task-specific high-level representations and capturing global relationships through the attention mechanism; by combining the advantages of both, the accuracy of prediction can be enhanced.

[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A salient object detection method based on a brain-inspired dual-process CNN-Transformer network, characterized in that: The following steps are involved: S1: construct a dataset for salient object detection, and divide the dataset into a training set, a test set, and a validation set; S2: constructing a network model based on pyramid attention U-Net and RECA-Transformer, wherein the network model based on pyramid attention U-Net and RECA-Transformer includes a dual-branch pyramid U-Net module and a residual efficient channel attention converter module; wherein the dual-branch pyramid U-Net module is an encoder-decoder architecture including an encoder part and a decoder part; The encoder part of the dual-branch pyramid U-Net module includes a pre-trained VGG16 network and a spatial attention module; the pre-trained VGG16 network is used to gradually extract feature information of the input image, and at the same time, the spatial attention module is combined to form a pyramid structure, so as to more effectively extract the spatial feature representation X and visual feature V of the input image, and the spatial feature representation X is enhanced in combination with the visual feature V; The decoder part of the dual-branch pyramid U-Net module uses transposed upsampling, linear interpolation upsampling, spatial context pooling module and residual coordinate attention module; the transposed upsampling, linear interpolation upsampling and spatial context pooling modules are used interchangeably to retain the spatial information of the feature map; the residual coordinate attention module is used to generate a spatial attention feature map; The residual efficient channel attention converter includes a RECA conversion block for feature extraction, which captures global characteristics through multi-layer attention mechanism and weight learning; S3: Use the training set to train the network model based on the pyramid attention U-Net and RECA-Transformer, use the validation set to adjust the model parameters, determine whether it is overfitting, use the combination of the basic loss function BaseLoss and the boundary attention loss function EdgeLoss as the model's attention boundary optimization loss function TotalLoss, and complete the training of the network model based on the pyramid attention U-Net and RECA-Transformer; use the test set to test the network performance of the network model based on the pyramid attention U-Net and RECA-Transformer; S4: Use the trained pyramid attention U-Net and RECA-Transformer based network models for salient object detection.

2. The method for salient object detection based on the brain-inspired dual-process CNN-Transformer network according to claim 1, characterized in that: The encoder part of the dual-branch pyramid U-Net module includes five stages, namely, the first stage consisting of the 2nd to 5th layers of the VGG16 network and the spatial attention module, the second stage consisting of the 5th to 10th layers of the VGG16 network and the spatial attention module, the third stage consisting of the 10th to 17th layers of the VGG16 network and the spatial attention module, the fourth stage consisting of the 17th to 24th layers of the VGG16 network and the spatial attention module, and the fifth stage consisting of the 24th to 31st layers of the VGG16 network and the spatial attention module; wherein the VGG16 network uses pre-trained convolutional layers and pooling layers to extract feature information of the image, and in each convolution stage, the size of the feature map is successively reduced to 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image; the spatial attention module consists of a convolutional layer and an effective channel attention module, the convolutional layer in the spatial attention module is used to extract the spatial feature representation X of the feature map, and the effective channel attention module is used to enhance the spatial feature representation.

3. The method for salient object detection based on the brain-inspired dual-process CNN-Transformer network according to claim 2, characterized in that: The expression of the convolutional layer in the spatial attention module is as follows: X=(Conv2D 3×3 (ReLU(Conv2D 1×1 (x c )))), Among them, X represents the spatial feature representation of the feature map, Conv2D 3×3 (·) represents a two-dimensional convolution, the convolution kernel size is 3×3, ReLU (·) represents the ReLU activation function, Conv2D 1×1 (·) represents a two-dimensional convolution, the convolution kernel size is 1×1, x c Represents the input feature map of the spatial attention module; The effective channel attention module includes a global average pooling layer GAP, a one-dimensional convolutional layer and a Sigmoid activation function stacked in sequence, and the output of the Sigmoid activation function is multiplied by the input of the global average pooling layer along the channel as the output of the effective channel attention module; the expression of the effective channel attention module is as follows: X′=ECA(X)=X·σ(Conv1D 1×1 (AvgPool(X))), Among them, X′ represents the features after enhancement processing, ECA(·) represents the processing function of the effective channel attention module, σ(·) refers to the Sigmoid activation of the result of one-dimensional convolution to obtain the channel attention weight, Conv1D 1×1 (·) represents one-dimensional convolution, the convolution kernel size is 1×1, and AvgPool(·) represents the global average pooling operation; That is, the expression of the spatial attention module is as follows: X′=ECA(Conv2D 3×3 (ReLU(Conv2D 1×1 (x c ))))。 4. The method for salient object detection based on a brain-inspired dual-process CNN-Transformer network according to claim 1, characterized in that: The decoder part of the dual-branch pyramid U-Net module includes six stages. The first stage of the decoder part includes a residual coordinate attention module and a pixel reconstruction module; the second stage of the decoder part includes linear interpolation upsampling, spatial context pooling module, residual coordinate attention module, slicing and splicing operations; the third, fourth and fifth stages of the decoder part all include transposition upsampling, linear interpolation upsampling, spatial context pooling module, residual coordinate attention module, slicing and splicing operations; the sixth stage of the decoder part includes transposition upsampling, slicing and splicing operations.

5. The method for salient object detection based on a brain-inspired dual-process CNN-Transformer network according to claim 1, characterized in that: The spatial context pooling module first uses a 1×1 convolution operation to map the input feature map x of the spatial context pooling module to a single-channel feature map G, and the dimension of the obtained feature map G is (N, 1, H, W). Subsequently, the H channel and the W channel of the feature map G are combined together, and the H and W dimensions of the feature map G are merged to obtain a feature map G′, and the dimension of the feature map G′ is (N, 1, H×W). Then, a Softmax operation is performed to calculate the weight of each position of the feature map G′, and the feature map G′ after channel merging is normalized and applied to the input feature map x, and an enhanced feature map is obtained by element-wise multiplication.

6. The method for salient object detection based on the brain-inspired dual-process CNN-Transformer network according to claim 1, characterized in that: The residual coordinate attention module includes a parallel horizontal global average pooling layer and a vertical global average pooling layer, a splicing operation, a 3×3 convolution layer, a batch normalization layer, and a nonlinear activation layer, followed by two parallel 1×1 convolution layers and two parallel Sigmoid activation functions. The outputs of the two Sigmoid activation functions are expanded to the size of the input feature map, and then the expanded feature map is combined with the input feature map of the residual coordinate attention module through element-by-element multiplication, and the obtained output is added to the input feature map of the residual coordinate attention module through a residual connection to complete the channel expansion of the input feature map; The process of the residual coordinate attention module for channel expansion of the input feature map is as follows: first, the input feature map of the residual coordinate attention module is subjected to global average pooling in the horizontal and vertical directions respectively through the parallel horizontal global average pooling layer and the vertical global average pooling layer to extract the global average information and retain the spatial information; then, the horizontal and vertical pooled feature maps are concatenated in the second dimension to generate a new feature map for further processing and analysis; the expression is as follows: y=concat(x h ,x w ,dim=2); Among them, x h and x w They represent the pooling results in the vertical and horizontal directions respectively, concat(·) represents the concatenation operation, and y represents the concatenated feature map; Then, a 3×3 convolutional layer and a batch normalization layer are used to reduce the dimension of the concatenated feature map, and the nonlinear characteristics are introduced through the ReLU activation function. The expression is as follows: and′=ReLU(BN(Conv2D 3×3 (and))); Among them, y′ represents the feature map after dimensionality reduction, ReLU(·) represents the ReLU activation function, BN(·) represents normalization, Conv2D 3×3 (·) represents a two-dimensional convolution with a kernel size of 3×3; Next, the reduced feature map y′ is separated into horizontal feature maps x w1 and vertical feature map x h1 , and generate attention weights through 1×1 convolution layer and Sigmoid activation function respectively, the expression is as follows: x h1 ,x w1 =split(y′,[h,w],dim=2); x h2 =Sigmoid(Conv2D 1×1 (x h1 )); x w2 =Sigmoid(Conv2D 1×1 (x w1 )); Among them, split(·) means splitting the feature map in the second dimension, Sigmoid(·) means the sigmoid activation function, x h2 represents the attention weight vertical feature map, x w2 Represents the attention weight level feature map; Then, the generated attention weight vertical feature map x h2 and the attention weight level feature map x w2 Expand to the size of the input feature map and combine them with the input feature map through element-wise multiplication, as follows: x h3 =x h2 ·expand(h,w); x w3 =x w2 ·expand(h,w); f=identity·x h3 ·x w3 ; Among them, f represents the output result after element-by-element multiplication, expand(h,w) represents the length and width of the attention weight feature map expanded to h and w, identity represents the initial input feature map of the residual coordinate attention module, and x h3 represents the expanded vertical feature map, x w3 represents the expanded horizontal feature map; finally, the input and output features are combined through the residual connection.

7. The method for salient object detection based on a brain-inspired dual-process CNN-Transformer network according to claim 1, characterized in that: The residual efficient channel attention converter module first performs preliminary processing on the input data through the feature block partitioning module. The input data is divided into small blocks, and each small block of input data is converted into a feature vector representation through a linear embedding module; the converted feature vector representation is sequentially processed by multiple continuous RECA conversion blocks and linear embedding modules; then, the spatial resolution of the feature map is reduced through the feature block merging module, and feature extraction is performed through two single RECA conversion blocks at the top of the model; The extracted data features are adjusted in resolution through the block expansion module, gradually increasing the spatial resolution of the feature map to achieve upsampling, thereby generating high-resolution output features; It is processed step by step again through multiple continuous RECA conversion blocks and linear embedding modules. The continuous RECA conversion block at this stage will receive the feature information transmitted across layers by the continuous RECA conversion block at the front end. The extracted data features are processed step by step again and then the spatial resolution of the feature map is gradually reduced through the feature block merging module, while the channel dimension is increased to achieve efficient feature extraction and calculation optimization. Finally, linear projection is used to adjust the channel dimension of the feature map as output to obtain the final feature map.

8. The method for salient object detection based on a brain-inspired dual-process CNN-Transformer network according to claim 1, characterized in that: The continuous RECA conversion block includes two connected RECA conversion blocks, wherein the RECA conversion block includes layer normalization, window-based multi-head self-attention mechanism, layer normalization, residual MLP module and effective channel attention module connected in sequence, wherein the input of the first layer normalization is added to the output of the window-based multi-head self-attention mechanism as the input of the second layer normalization, and the input of the second layer normalization is added to the output of the effective channel attention module as the output of the RECA conversion block; in the continuous RECA conversion block, the output of the first RECA conversion block is used as the input of the second RECA conversion block.

9. The method for salient object detection based on the brain-inspired dual-process CNN-Transformer network according to claim 8, characterized in that: The residual MLP module includes an effective channel attention module, a fully connected layer fc1, a fully connected layer fc2, and a fully connected layer fc3 which are connected in sequence, and the output of the fully connected layer fc1 is added to the output of the fully connected layer fc2 through a residual connection and used as the input of the fully connected layer fc3.

10. The method for salient object detection based on a brain-inspired dual-process CNN-Transformer network according to claim 1, characterized in that: The calculation formula of the attention boundary optimization loss function TotalLoss composed of the basic loss function BaseLoss and the boundary attention loss function EdgeLoss is as follows: TotalLoss=BaseLoss+β·EdgeLoss; AttentionWeights=1+α·Edges; Among them, BaseLoss is the basic loss, which calculates the cross entropy loss between the output image predicted by the model and the true label. The edge map edges is generated by calculating the gradient components of the true mask map in the vertical and horizontal directions to obtain a high-quality edge map. β is the weight of the edge loss, EdgeLoss is the edge optimization loss, and CrossEntropy(p i ,t i ) represents the cross entropy loss function, p i represents the predicted value, t i Represents the actual value, N represents a total of N points, AttentionWeights represents the attention weight, α represents the proportion of edge weight, and Edges represents the edge.

Images