Common feature saliency target detection method based on lightweight network

Abstract

The invention discloses a common feature saliency target detection method based on a lightweight network, and the method comprises the steps: constructing a training set which comprises a color visible light image, an infrared image and a depth image, and building a lightweight neural network; the lightweight neural network is mainly composed of a common structure extraction module, a bimodal fusion module and a multi-scale feature refining module. Inputting the plurality of pairs of color visible light images in the training set and the infrared images and the depth images corresponding to the color visible light images into a neural network for multiple times of network training, and obtaining a neural network training model after the network training is finished; and the neural network training model is used to predict the test image pair to obtain the saliency target image, so that the accuracy of saliency target detection can be effectively improved under the condition of low parameter quantity and calculation quantity.

Description

Technical Field

[0001] This invention relates to the field of salient target detection technology, and in particular to a method for salient target detection based on shared features using lightweight networks. Background Technology

[0002] Salient object detection aims to capture and segment the regions or objects in images or videos that are most attractive to human attention. As an important preprocessing step, salient object detection has been widely applied in computer vision and image processing tasks such as image segmentation, object tracking, image retrieval, and image quality assessment. In recent years, convolutional neural networks and deep learning have pushed the performance of salient object detection to new heights due to their powerful learning capabilities and excellent performance in feature extraction. However, in some extreme scenarios (such as low-light and cluttered scenes), it is often difficult to extract valuable information from color visible light images alone, which affects the effectiveness of salient object detection models under complex conditions. Unlike color visible light images, thermal infrared cameras can capture thermal information that is less affected by environmental factors such as darkness and inclement weather, thus reflecting temperature differences, boundaries, and geometry. Therefore, deploying thermal infrared devices to collect thermal information and using color visible light and infrared image salient object detection models to record objects can improve the model's perception capabilities in more scenarios. Depth images contain geometric information about objects, and in complex color scenes, the spatial information of different objects can better help the model perceive object information in the scene. Therefore, in recent years, research on combining color visible light images with infrared images and depth images for salient target detection has received increasing attention.

[0003] On the other hand, existing visible-depth-infrared (RGB-DT) salient target detection models have made significant breakthroughs in accuracy, but these high-precision models usually require high computational costs and large model sizes to process multimodal information. This means that they can only be deployed in graphics processing unit facilities with high performance requirements but limited computing power. They cannot meet the needs of some practical scenarios in terms of computational latency and bandwidth costs, such as autonomous driving and monitoring IoT.

[0004] Currently, several lightweight RGB-DT salient object detection models exist, primarily based on convolutional neural networks (CNNs). These methods typically employ a lightweight backbone network to extract features from three modalities, then perform multimodal, multi-level feature fusion through carefully designed lightweight modules. While these methods successfully compress model size and can be deployed on resource-constrained devices, there is still room for improvement in model size, and the accuracy remains insufficient. Therefore, how to enhance the predictive performance of lightweight RGB-DT salient object detection models while maintaining a low model size has become a pressing issue. Summary of the Invention

[0005] To address the aforementioned shortcomings, this invention proposes a common feature salient target detection method based on lightweight networks, which can effectively improve the accuracy of salient target detection with low parameter and computational cost.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a common feature salient target detection method based on a lightweight network, comprising the following steps: first, constructing a three-modal training set containing multiple pairs of registered color visible light images, infrared images, and depth images, and building a lightweight neural network; then, inputting the three-modal images in the training set into the lightweight neural network for multiple rounds of network training, obtaining a neural network training model after the network training is completed; and then using the neural network training model again to predict the test image pairs, predicting the salient target images of the test image pairs, wherein the test image pairs include a color visible light image and its corresponding infrared image and depth image; The lightweight neural network consists of a feature extraction module, a common structure extraction module, a dual-modal fusion module, and a multi-scale feature refinement module connected in sequence. The feature extraction module includes a first MobileNetV2 backbone network for extracting feature information and scale information from color visible light images, and a second MobileNetV2 backbone network for extracting feature information and scale information from infrared images and depth images; The common structure extraction block is used to perform channel max pooling on the input depth feature map and infrared feature map, and then generate common structure weights through element-wise multiplication and concatenation operations. Finally, it outputs a geometrically aligned common structure feature map using depthwise convolution and residual connections. The bimodal fusion block is used to perform global average pooling on color visible light features and common structural features, and then generate modal attention weights through a fully connected layer; the bimodal features are weighted, fused, and output; The feature refinement block receives features from the dual-modal fusion module, the previous refinement block, and the common structure extraction module. It extracts multi-scale context using dilated convolutions and depthwise convolutions with different dilation rates. It outputs the salient feature map at the current scale, where the fifth-level output is the final salient target image.

[0007] As an improvement, the first MobileNetV2 backbone network performs five-layer cascaded feature extraction on the input color visible light image. Each layer receives the output of the previous layer and generates a corresponding color feature map. The first to fifth layers output the first color visible light feature map, respectively. Second color visible light characteristic map Third color visible light characteristic map Fourth color visible light characteristic map Fifth color visible light characteristic map The second MobileNetV2 backbone network uses a dual-branch processing method to perform five-layer cascaded feature extraction on the input infrared image. Each layer receives the output of the previous layer and generates a corresponding infrared image feature map. The first to fifth layers output the first infrared image feature map respectively. Second infrared image feature map Third infrared image feature map Fourth infrared image feature map Fifth infrared image feature map Simultaneously, a five-layer concatenated feature extraction process is performed on the input infrared image. Each layer sequentially receives the output of the previous layer and generates a corresponding depth image feature map. The first to fifth layers respectively output the first depth image feature map. Second depth image feature map Third-depth image feature map Fourth depth image feature map Fifth depth image feature map .

[0008] As an improvement, the common structure extraction module includes five structurally identical common structure extraction blocks used to extract common geometric structure information in infrared and depth images, realizing cross-modal complementary fusion. The common structure extraction block consists of a first channel max pooling layer, a second channel max pooling layer, a first 1×1 convolutional layer, a first sigmoid activation layer, a first 3×3 convolutional layer, a first BN layer, a second BN layer, a first ReLU activation layer, a second ReLU activation layer, and a first 3×3 depth convolutional layer.

[0009] As an improvement, the first The implementation process of the shared structure extraction block is as follows: The first input terminal of the common structure extraction block receives Second input terminal receives Third input terminal receives The input terminal of the first channel max pooling layer receives... The feature map output by the output terminal The input of the second channel maximum pooling layer receives... The feature map output by the output terminal ;Will and Element-wise multiplication yields the feature map. ;Will and Element-wise multiplication yields the feature map. ;Will and By stitching along the channels, a feature map is obtained. The receiver of the first 1×1 convolutional layer receives... The receiver of the first sigmoid activation layer receives the feature map output by the output of the first 1×1 convolutional layer, and the output of the first sigmoid activation layer outputs the feature map. ;Will , and By stitching along the channels, a feature map is obtained. The input of the first 3×3 convolutional layer receives... The input of the first Batch Normalization (BN) layer receives the feature map output from the first 3×3 convolutional layer, and the input of the first ReLU activation layer receives the feature map output from the first BN layer. The output of the first ReLU activation layer also receives the feature map. ;Will and By stitching along the channels, a feature map is obtained. ;Will and Element-wise addition yields the feature map. The input of the first 3×3 depth convolutional layer receives... The input of the second BN layer receives the feature map output from the first 3×3 depth convolutional layer, and the input of the second ReLU activation layer receives the feature map output from the second BN layer. ;in, , ,and and One-to-one correspondence; for the first common structure extraction block, the third receiving end has no input, at this time and They are exactly the same, where DT represents both depth and infrared image modes.

[0010] As an improvement, the bimodal fusion module includes five structurally identical bimodal fusion blocks for fusion strategies targeting semantic information, addressing the problem of high-quality fusion under trimodal conditions. The bimodal fusion block consists of a first global average pooling layer, a second global average pooling layer, a second 1×1 convolutional layer, a third 1×1 convolutional layer, a fourth 1×1 convolutional layer, a fifth 1×1 convolutional layer, a sixth 1×1 convolutional layer, a seventh 1×1 convolutional layer, a third ReLU activation layer, a fourth ReLU activation layer, a fifth ReLU activation layer, a second sigmoid activation layer, a third sigmoid activation layer, a fourth sigmoid activation layer, a fifth sigmoid activation layer, a third channel max pooling layer, a fourth channel max pooling layer, a first channel average pooling layer, a second channel average pooling layer, a second 3×3 depthwise convolutional layer, and a third batch normalization (BN) layer.

[0011] As an improvement, the first The implementation process of the dual-modal fusion block is as follows: The first input of the dual-modal fusion block receives The second receiving end receives The third input terminal receives ;Will The feature map is obtained by sequentially passing through a first global average pooling layer, a second 1×1 convolutional layer, a third ReLU activation layer, a third 1×1 convolutional layer, and a second sigmoid activation layer. ;Will The feature map is obtained by sequentially passing through the second global average pooling layer, the fourth 1×1 convolutional layer, the fourth ReLU activation layer, the fifth 1×1 convolutional layer, and the third sigmoid activation layer. ;Will and Element-wise multiplication yields the feature map. ;Will and Element-wise addition yields the feature map. ;Will and Element-wise multiplication yields the feature map. ;Will and Element-wise addition yields the feature map. ;Will After passing through the third channel max pooling layer, the feature map is obtained. ;Will After passing through the first channel average pooling layer, the feature map is obtained. ;Will and By stitching along the channels, a feature map is obtained. ;Will The feature map is obtained by sequentially passing through the sixth 1×1 convolutional layer and the fourth sigmoid activation layer. ;Will and Element-wise multiplication yields the feature map. ;Will After passing through the fourth channel max pooling layer, the feature map is obtained. ;Will After passing through the second channel average pooling layer, the feature map is obtained. ;Will and By stitching along the channels, a feature map is obtained. ;Will The feature map is obtained by sequentially passing through the seventh 1×1 convolutional layer and the fifth sigmoid activation layer. ;Will and Element-wise multiplication yields the feature map. ;Will and By stitching along the channels, a feature map is obtained. ;Will and By stitching along the channels, a feature map is obtained. ;Will The feature map is obtained by sequentially passing the second 3×3 depth convolutional layer, the third BN layer, and the fifth ReLU activation layer. ;in, , ,and and One-to-one correspondence; for the first dual-mode fusion block, the third receiver has no input, at this time and Exactly the same.

[0012] As an improvement, the multi-scale feature refinement module includes five structurally identical feature refinement blocks for efficiently transferring and fusing semantic and detailed information across different scales and layers. Each feature refinement block consists of an eighth 1×1 convolutional layer, a ninth 1×1 convolutional layer, a tenth 1×1 convolutional layer, an eleventh 1×1 convolutional layer, a twelfth 1×1 convolutional layer, a thirteenth 1×1 convolutional layer, a third 3×3 depth convolutional layer, a fourth 3×3 depth convolutional layer, a fifth 3×3 depth convolutional layer, a first dilated convolutional layer with a dilation rate of 3, a second dilated convolutional layer with a dilation rate of 5, a third dilated convolutional layer with a dilation rate of 7, a second 3×3 convolutional layer, a fourth BN layer, a sixth ReLU activation layer, a seventh ReLU activation layer, a first channel attention layer, and a first upsampling layer.

[0013] As an improvement, the first The implementation process of the feature refining block is as follows: The first input of the feature refining block receives The second input terminal receives The third input terminal receives ;Will , , Element-wise addition yields the feature map. ;Will The inputs are fed into the eighth, ninth, tenth, eleventh, and twelfth 1×1 convolutional layers, respectively, to obtain the corresponding feature maps. Feature map Feature map Feature map Feature map ;Will and Perform element-wise addition to obtain the feature map. ;Will The feature map is obtained by sequentially passing through a third 3×3 depth convolutional layer and a first dilated convolutional layer. ;Will and Perform element-wise addition to obtain the feature map. ;Will The feature map is obtained by sequentially passing through a fourth 3×3 depth convolutional layer and a second dilated convolutional layer. ;Will and Perform element-wise addition to obtain the feature map. ;Will The feature map is obtained by sequentially passing through a fifth 3×3 depth convolutional layer and a third dilated convolutional layer. ;Will , , , By stitching along the channels, a feature map is obtained. ;Will The feature map is obtained by sequentially passing through the second 3×3 convolutional layer, the fourth BN layer, the sixth ReLU activation layer, and the first channel attention layer. ;Will and Element-wise addition yields the feature map. The input of the seventh ReLU activation layer receives... The feature map output from the seventh ReLU activation layer is also the feature map of the seventh ReLU activation layer. Feature map of the first output of the multi-scale feature extraction block. ;Will The feature map obtained after the thirteenth 1×1 convolutional layer and the first upsampling layer is also the... Feature map at the second output of the multi-scale feature extraction block ;in, , ,and and One-to-one correspondence; for the fifth feature refinement block, the third receiver has no input, at this time only for , When adding elements, where, As the final salient target image, RDT represents three modes: color visible light, depth, and infrared.

[0014] As an improvement, the process of obtaining the neural network training model is as follows: each pair of color visible light images and their corresponding infrared and depth images from the training set are input into the neural network for network training, and the loss function is calculated before the end of each round of network training. To optimize the neural network, a trained neural network model is obtained after a total of 190-210 rounds of network training: in, This represents the salient target image output by the multi-scale feature extraction module of the neural network. Represents a label image. This represents the weighted binary cross-entropy loss. This indicates the weighted average loss.

[0015] As an improvement, a neural network-trained model is used to predict the salient target image of the test image pair. The process is as follows: First, arbitrarily select a pair of original color visible light images and their corresponding original infrared and depth images; then, downsample the pair of original color visible light images, original infrared images, and original depth images. The size of the downsampled image is... The test image pairs are then input into the trained neural network model to predict the corresponding salient target images.

[0016] Compared with the prior art, the advantages of the present invention are as follows: This invention significantly improves perception capabilities in complex environments (such as low-light and cluttered scenes) by fusing trimodal information from color visible light images, infrared images, and depth images. Infrared images can capture thermal information unaffected by environmental factors such as darkness and inclement weather, while depth images provide geometric structural information of objects, helping to perceive objects more accurately in complex color scenes.

[0017] This invention employs a lightweight neural network, including a lightweight backbone network (such as MobileNetV2) and carefully designed lightweight modules (such as a common structure extraction module, a bimodal fusion module, and a multi-scale feature refinement module). Based on common structural features in infrared and depth images, it extracts and optimizes structural features using the same backbone network, simplifying modal representation and reducing the number of backbone networks required for modal feature extraction. This reduces the number of model parameters and improves model speed, resulting in a smaller model size while maintaining high performance, lowering computational costs, and facilitating deployment on resource-constrained devices, thus expanding the application scope of salient object detection technology. Furthermore, this invention designs a bimodal fusion module for semantic information and a multi-scale feature refinement module during trimodal interaction and fusion, enabling efficient transfer and fusion of semantic and detailed information across different scales and levels, supplemented by a saliency supervision strategy. This combination strengthens the unique features of each modality while deeply mining shared information, significantly improving the overall performance of multimodal salient object detection. Simultaneously, the shared structure extraction module extracts and aligns shared geometric structure information between depth and infrared images. This module fully leverages the advantages of both modalities being insensitive to illumination and having consistent geometry, and combines the texture semantics of the RGB branches to achieve cross-modal complementary fusion, thereby effectively improving the accuracy and robustness of salient target detection.

[0018] This invention employs specific loss functions (such as weighted binary cross-entropy loss and weighted cross-union loss) for network training. These loss functions help optimize the performance of the neural network and improve the accuracy of salient object detection. By constructing a trimodal training set containing multiple registration pairs, this invention can more fully utilize the training data and improve the model's generalization ability. By combining multimodal information and optimizing the network structure, this invention can achieve high-precision salient object detection. This helps improve the performance of related applications and the user experience. Attached Figure Description

[0019] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Figure 1 This is a diagram illustrating the overall implementation framework of the method of the present invention; Figure 2 A schematic diagram illustrating the structural composition of the neural network constructed using the method of this invention; Figure 3 A schematic diagram of the composition structure of the common structure extraction block in the neural network constructed by the method of the present invention; Figure 4 A schematic diagram illustrating the structural composition of the dual-modal fusion block in the neural network constructed by the method of this invention; Figure 5 This is a schematic diagram of the composition structure of the feature refinement block in the neural network constructed by the method of the present invention. Detailed Implementation

[0020] like Figures 1 to 5 As shown, a common feature saliency target detection method based on a lightweight network includes the following steps: First, a three-modal training set containing multiple pairs of registered color visible light images, infrared images, and depth images is constructed, and a lightweight neural network is built; then, the three-modal images in the training set are input into the lightweight neural network for multiple rounds of network training, and a neural network training model is obtained after the network training is completed; and the neural network training model is used again to predict the test image pairs to obtain the saliency target images of the test image pairs, where the test image pairs include a color visible light image and its corresponding infrared image and depth image; The training set is constructed as follows: at least 1000 pairs of original color visible light images and their corresponding original infrared and depth images are selected; then, each original color visible light image and its corresponding original infrared and depth images are downsampled, and the size of the downsampled image is [size missing]. Then all sizes are The training set consists of color visible light images and their corresponding infrared and depth images; among them, 1048 pairs of original color visible light images and their corresponding original infrared and depth images are selected. .

[0021] The lightweight neural network consists of a feature extraction module, a common structure extraction module, a dual-modal fusion module, and a multi-scale feature refinement module connected in sequence. The feature extraction module includes a first MobileNetV2 backbone network for extracting feature information and scale information from color visible light images, and a second MobileNetV2 backbone network for extracting feature information and scale information from infrared images and depth images; The first layer of the first MobileNetV2 backbone network extracts features from the color visible light image and outputs the first color visible light feature map. The second layer receiver of the first MobileNetV2 backbone network It also outputs a second color visible light feature map. The third-layer receiver of the first MobileNetV2 backbone network It also outputs a third-color visible light feature map. The third layer, the fourth layer receiver of the first MobileNetV2 backbone network. It also outputs the fourth color visible light feature map. The fifth layer receiver of the first MobileNetV2 backbone network It also outputs the fifth color visible light feature map. The first layer of the second MobileNetV2 backbone network extracts features from the infrared image and outputs the first infrared image feature map. The second layer receiver of the second MobileNetV2 backbone network And output the second infrared image feature map The third layer receiver of the second MobileNetV2 backbone network And output the third infrared image feature map The fourth layer receiver of the second MobileNetV2 backbone network And output the fourth infrared image feature map The fifth layer receiver of the second MobileNetV2 backbone network And output the fifth infrared image feature map Furthermore, the first layer of the second MobileNetV2 backbone network extracts features from the depth image and outputs the first depth image feature map. The second layer receiver of the second MobileNetV2 backbone network And output the second depth image feature map. The third layer receiver of the second MobileNetV2 backbone network And output the third depth image feature map The fourth layer receiver of the second MobileNetV2 backbone network It also outputs the fourth depth image feature map. The fifth layer receiver of the second MobileNetV2 backbone network It also outputs the fifth depth image feature map. ; The common structure extraction module includes five structurally identical common structure extraction blocks used to extract common geometric structure information from infrared and depth images, achieving cross-modal complementary fusion; the first input terminal of the first common structure extraction block receives... The second input terminal receives The third receiving end has no input, and the feature map output by the output end of the first common structure extraction block is denoted as... The first input terminal of the second common structure extraction block receives... The second input terminal receives The third input terminal receives The feature map output from the output end of the second common structure extraction block is denoted as... The first input terminal of the third common structure extraction block receives... The second input terminal receives The third input terminal receives The feature map output by the third common structure extraction block is denoted as... The first input terminal of the fourth common structure extraction block receives... The second input terminal receives The third input terminal receives The feature map output by the fourth common structure extraction block is denoted as... The first input terminal of the fifth common structure extraction block receives... The second input terminal receives The third input terminal receives The feature map output by the fifth common structure extraction block is denoted as... DT represents both depth and infrared image modes; The bimodal fusion module comprises five structurally identical bimodal fusion blocks that differ only in their inputs and outputs, designed for fusion strategies targeting semantic information. This addresses the issue of high-quality fusion in trimodal environments. The first input of the first bimodal fusion block receives... The second input terminal receives The third receiver has no input, and the feature map output by the output of the first dual-mode fusion block is denoted as... The first input terminal of the second dual-modal fusion block receives... The second input terminal receives The third input terminal receives The feature map output from the output terminal of the second bimodal fusion block is denoted as... The first input terminal of the third dual-modal fusion block receives... The second input terminal receives The third input terminal receives The feature map output from the output terminal of the third bimodal fusion block is denoted as... The first input terminal of the fourth dual-mode fusion block receives... The second input terminal receives The third input terminal receives The feature map output from the output terminal of the fourth bimodal fusion block is denoted as... The first input terminal of the fifth dual-modal fusion block receives... The second input terminal receives The third input terminal receives The feature map output from the output terminal of the fifth bimodal fusion block is denoted as... ; where RT represents two modes: color visible light and infrared image; The multi-scale feature refinement module includes five structurally identical feature refinement blocks that differ only in their inputs and outputs. These blocks are used to efficiently transfer and fuse semantic and detailed information across different scales and levels. The first input of the first feature refinement block receives... The second input terminal receives The third input terminal receives The feature map output from the first output terminal of the first feature refining block is denoted as... The feature map output by the second output terminal is denoted as The first input terminal of the second feature refining block receives... The second input terminal receives The third input terminal receives The feature map output from the first output terminal of the second feature refining block is denoted as... The feature map output by the second output terminal is denoted as The first input terminal of the third feature refining block receives... The second input terminal receives The third input terminal receives The feature map output from the first output terminal of the third feature refining block is denoted as... The feature map output by the second output terminal is denoted as The first input terminal of the fourth feature refining block receives... The second input terminal receives The third input terminal receives The feature map output from the first output terminal of the fourth feature refining block is denoted as... The feature map output by the second output terminal is denoted as The first input terminal of the fifth feature refining block receives... The second input terminal receives The third input terminal has no input, and the feature map output from the first output terminal of the fifth feature refining block is denoted as... The feature map output by the second output terminal is denoted as ,in, As the final salient target image, RDT represents three modes: color visible light, depth, and infrared. Here, the sizes of the color visible light image, infrared image, and depth image are all... In this embodiment Take 320×320, , and The size is , , and The size is , , and The size is , , and The size is , , and The size is , , and The size is , , and The size is , , and The size is , , and The size is , , and The size is , , , , and The size is ; The common structure extraction block is used to perform channel max pooling on the input depth feature map and infrared feature map; generate common structure weights through element-wise multiplication and concatenation operations; and output a geometrically aligned common structure feature map using depthwise convolution and residual connections. The common structure extraction block consists of a first channel max pooling layer, a second channel max pooling layer, a first 1×1 convolutional layer, a first sigmoid activation layer, a first 3×3 convolutional layer, a first BN layer, a second BN layer, a first ReLU activation layer, a second ReLU activation layer, and a first 3×3 depthwise convolutional layer. No. The implementation process of the shared structure extraction block is as follows: The first input terminal of the common structure extraction block receives Second input terminal receives Third input terminal receives The input terminal of the first channel max pooling layer receives... The feature map output by the output terminal is denoted as The input of the second channel maximum pooling layer receives... The feature map output by the output terminal is denoted as ;Will and Element-wise multiplication yields a feature map, denoted as . ;Will and Element-wise multiplication yields a feature map, denoted as . ;Will and The feature map is obtained by splicing along the channel. The receiver of the first 1×1 convolutional layer receives... The receiver of the first sigmoid activation layer receives the feature map output from the output of the first 1×1 convolutional layer. The feature map output from the first sigmoid activation layer is denoted as... ;Will , and The feature map is obtained by splicing along the channel. The input of the first 3×3 convolutional layer receives... The input of the first Batch Normalization (BN) layer receives the feature map output from the first 3×3 convolutional layer, and the input of the first ReLU activation layer receives the feature map output from the first BN layer. The feature map output from the first ReLU activation layer is denoted as... ;Will and The feature map is obtained by splicing along the channel. ;Will and Element-wise addition yields the feature map. The input of the first 3×3 depth convolutional layer receives... The input of the second BN layer receives the feature map output from the first 3×3 depth convolutional layer, and the input of the second ReLU activation layer receives the feature map output from the second BN layer. The feature map output from the second ReLU activation layer is denoted as... ;in, , ,and and One-to-one correspondence; for the first common structure extraction block, the third receiving end has no input, at this time and They are exactly the same; here, element-wise multiplication, concatenation along channels, and element-wise addition are all common operations in neural networks; channel max pooling layers are common modules in neural networks; The bimodal fusion block is used to perform global average pooling on color visible light features and common structural features; generate modal attention weights through fully connected layers; and weightedly fuse bimodal features and output them. The bimodal fusion block consists of a first global average pooling layer, a second global average pooling layer, a second 1×1 convolutional layer, a third 1×1 convolutional layer, a fourth 1×1 convolutional layer, a fifth 1×1 convolutional layer, a sixth 1×1 convolutional layer, a seventh 1×1 convolutional layer, a third ReLU activation layer, a fourth ReLU activation layer, a fifth ReLU activation layer, a second sigmoid activation layer, a third sigmoid activation layer, a fourth sigmoid activation layer, a fifth sigmoid activation layer, a third channel max pooling layer, a fourth channel max pooling layer, a first channel average pooling layer, a second channel average pooling layer, a second 3×3 depthwise convolutional layer, and a third batch normalization (BN) layer. No. The implementation process of the dual-modal fusion block is as follows: The first input of the dual-modal fusion block receives The second receiving end receives The third input terminal receives The receiver of the first global average pooling layer receives... ,second The receiver of the convolutional layer receives the feature map output from the first global average pooling layer; the receiver of the third ReLU activation layer receives the feature map output from the second 1×1 convolutional layer; the receiver of the third 1×1 convolutional layer receives the feature map output from the third ReLU activation layer; the receiver of the second sigmoid activation layer receives the feature map output from the third 1×1 convolutional layer; and the feature map output from the second sigmoid activation layer is denoted as... The receiver of the second global average pooling layer receives... ,fourth The receiver of the convolutional layer receives the feature map output from the second global average pooling layer; the receiver of the fourth ReLU activation layer receives the feature map output from the fourth 1×1 convolutional layer; the receiver of the fifth 1×1 convolutional layer receives the feature map output from the fourth ReLU activation layer; and the receiver of the third sigmoid activation layer receives the feature map output from the fifth 1×1 convolutional layer. The feature map output from the third sigmoid activation layer is denoted as... ;Will and Element-wise multiplication yields a feature map, denoted as . ;Will and Element-wise addition yields the feature map. ;Will and Element-wise multiplication yields a feature map, denoted as . ;Will and Element-wise addition yields the feature map. The receiver of the third channel's maximum pooling layer receives... The feature map output from the third channel max pooling layer is denoted as... The receiver of the first channel average pooling layer receives... The feature map output from the first channel average pooling layer is denoted as... ;Will and The feature map is obtained by splicing along the channel. The receiver of the sixth 1×1 convolutional layer receives... The receiver of the fourth sigmoid activation layer receives the feature map output from the output of the sixth 1×1 convolutional layer. The feature map output from the fourth sigmoid activation layer is denoted as... ;Will and Element-wise multiplication yields a feature map, denoted as . The receiver of the fourth channel's maximum pooling layer receives... The feature map output from the fourth channel max pooling layer is denoted as... The receiver of the second channel average pooling layer receives... The feature map output from the first channel average pooling layer is denoted as... ;Will and The feature map is obtained by splicing along the channel. The receiver of the seventh 1×1 convolutional layer receives... The receiver of the fifth sigmoid activation layer receives the feature map output from the output of the seventh 1×1 convolutional layer. The feature map output from the fifth sigmoid activation layer is denoted as... ;Will and Element-wise multiplication yields a feature map, denoted as . ;Will and The feature map is obtained by splicing along the channel. ;Will and The feature map is obtained by splicing along the channel. The receiver of the second 3×3 depth convolutional layer receives... The receiving end of the third BN layer receives the feature map output from the output end of the second 3×3 depth convolutional layer, and the fifth ReLU activation layer receives the feature map output from the output end of the third BN layer. The feature map output from the output end of the fifth ReLU activation layer is denoted as... ;in, , ,and and One-to-one correspondence; for the first dual-mode fusion block, the third receiver has no input, at this time and They are exactly the same; here, global average pooling layer and channel average pooling layer are common modules in neural networks. The feature refinement block receives features from the dual-modal fusion module, the previous refinement block, and the common structure extraction module; it extracts multi-scale context using dilated convolutions and depthwise convolutions with dilation rates of 3, 5, and 7; and outputs the salient feature map at the current scale, with the fifth-level output being the final salient target image. The feature refinement block consists of the eighth 1×1 convolutional layer, the ninth 1×1 convolutional layer, the tenth 1×1 convolutional layer, the eleventh 1×1 convolutional layer, the twelfth 1×1 convolutional layer, the thirteenth 1×1 convolutional layer, the third 3×3 depth convolutional layer, the fourth 3×3 depth convolutional layer, the fifth 3×3 depth convolutional layer, the first dilated convolutional layer with an inflation rate of 3, the second dilated convolutional layer with an inflation rate of 5, the third dilated convolutional layer with an inflation rate of 7, the second 3×3 convolutional layer, the fourth BN layer, the sixth ReLU activation layer, the seventh ReLU activation layer, the first channel attention layer, and the first upsampling layer. No. The implementation process of the feature refining block is as follows: the first input terminal of the feature refining block receives... The second input terminal receives The third input terminal receives ;Will , , Element-wise addition yields the feature map. ;Will The inputs are fed into the eighth, ninth, tenth, eleventh, and twelfth 1×1 convolutional layers, respectively, to obtain the corresponding feature maps. Feature map Feature map Feature map Feature map ;Will and Perform element-wise addition to obtain the feature map. The input of the third 3×3 depth convolutional layer receives... The input of the first dilated convolutional layer with an inflation rate of 3 receives the feature map output from the output of the third 3×3 depth convolutional layer. The feature map output from the first dilated convolutional layer with an inflation rate of 3 is denoted as... ;Will and Perform element-wise addition to obtain the feature map. The input of the fourth 3×3 depth convolutional layer receives... The input of the second dilated convolutional layer with a dilation rate of 5 receives the feature map output from the output of the fourth 3×3 depth convolutional layer. The feature map output from the second dilated convolutional layer with a dilation rate of 5 is denoted as... ;Will and Perform element-wise addition to obtain the feature map. The input of the fifth 3×3 depth convolutional layer receives... The input of the third dilated convolutional layer with an inflation rate of 7 receives the feature map output from the output of the fifth 3×3 depth convolutional layer. The feature map output from the third dilated convolutional layer with an inflation rate of 7 is denoted as... ;Will , , , The feature map is obtained by splicing along the channel. The input of the second 3×3 convolutional layer receives... The input of the fourth BN layer receives the feature map output from the second 3×3 convolutional layer; the input of the sixth ReLU activation layer receives the feature map output from the fifth BN layer; and the input of the first channel attention layer receives the feature map output from the sixth ReLU activation layer. The feature map output from the first channel attention layer is denoted as... ;Will and Adding elements together yields a feature map, denoted as . The input of the seventh ReLU activation layer receives... The feature map output from the seventh ReLU activation layer is also the feature map of the seventh ReLU activation layer. The feature map at the first output of the multi-scale feature extraction block is denoted as... The input of the thirteenth 1×1 convolutional layer receives... The input of the first upsampling layer receives the feature map output from the output of the thirteenth 1×1 convolutional layer. The feature map output from the first upsampling layer is also the feature map of the thirteenth 1×1 convolutional layer. The feature map at the second output of the multi-scale feature extraction block is denoted as... ;in, , ,and and One-to-one correspondence; for the fifth feature refinement block, the third receiver has no input, at this time only for , Add elements together; The process of obtaining the neural network training model is as follows: each pair of color visible light images and their corresponding infrared and depth images from the training set are input into the neural network for network training. The loss function is calculated before the end of each round of network training. To optimize the neural network, a trained neural network model was obtained after a total of 200 rounds of network training: in, This represents the salient target image output by the multi-scale feature extraction module of the neural network. Represents a label image. This represents the weighted binary cross-entropy loss. Indicates the weighted average loss; The process of using a neural network-trained model to predict the salient target image of a test image pair is as follows: First, arbitrarily select a pair of original color visible light images and their corresponding original infrared and depth images; then, downsample the pair of original color visible light images, original infrared images, and original depth images. The size of the downsampled image is... The test image pairs are then input into the trained neural network model to predict the corresponding salient target images.

[0022] To further demonstrate the feasibility and effectiveness of the method of the present invention, experiments were conducted on the method of the present invention.

[0023] In the experiment, the method of this invention was used to test a dataset of salient objects in color visible light images, depth images, and infrared images (VDT2048 dataset). The VDT2048 dataset contains a training set and a test set. The training set includes 1048 pairs of color visible light images, infrared images, and depth images, i.e., 1048 training image pairs. The test set includes 1000 pairs of color visible light images, infrared images, and depth images, i.e., 1000 test image pairs.

[0024] In this embodiment, four commonly used objective parameters and three commonly used objective parameters for measuring model lightweightness are selected to evaluate the performance of the method of the present invention. These are S-measure, E-measure, F-measure, Mean Absolute Error (MAE), Params (M), Flops (G), and FPS. Table 1 shows the correlation between salient target images and label images obtained by the method of the present invention on the VDT2048 dataset, as well as the lightweightness of the model.

[0025] Table 1 shows the S-measure, E-measure, F-measure, and Mean Absolute Error (MAE) between salient target images and label images obtained using the method of this invention on the VDT2048 dataset. The results shown in Table 1 reveal that the method of this invention achieves high S-measure, E-measure, and F-measure, and low MAE on existing trimodal salient object detection datasets, while also exhibiting low parameter and computational complexity and high computational speed. This indicates that the salient object images and label images obtained by the method of this invention are quite similar, and the method can effectively complete the trimodal salient object detection task with low parameter and computational complexity.

[0026] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0027] The units described in some embodiments of this disclosure can be implemented in software or in hardware. The described units can also be located in a processor, and the functions described above can be performed at least in part by one or more hardware logic components.

[0028] The above description only illustrates the preferred embodiments of the present invention and should not be construed as limiting the scope of the claims. The present invention is not limited to the above embodiments, and variations in its specific structure are permitted. All modifications made within the scope of the independent claims of this invention are also within the scope of protection of this invention.

Claims

1. A common feature saliency target detection method based on lightweight networks, characterized in that, Includes the following steps: First, a trimodal training set containing multiple pairs of registered color visible light images, infrared images, and depth images is constructed, and a lightweight neural network is built. Then, the trimodal images in the training set are input into the lightweight neural network for multiple rounds of network training. After the network training is completed, a neural network training model is obtained. The neural network training model is then used again to predict test image pairs to obtain the salient target images of the test image pairs, which include color visible light images and their corresponding infrared and depth images. The lightweight neural network includes a feature extraction module, a common structure extraction module, a dual-modal fusion module, and a multi-scale feature refinement module connected in sequence. The feature extraction module includes a first MobileNetV2 backbone network for extracting feature information and scale information of color visible light images, and a second MobileNetV2 backbone network for extracting feature information and scale information of infrared images and depth images; The common structure extraction block is used to perform channel max pooling on the input depth feature map and infrared feature map, then generate common structure weights through element-wise multiplication and concatenation operations, and output geometrically aligned common structure feature map using depthwise convolution and residual connection. The dual-modal fusion block is used to perform global average pooling on color visible light features and common structural features, and then generate modal attention weights through a fully connected layer. Weighted fusion of dual-modal features and output; The feature refinement block receives features from the dual-modal fusion module, the previous refinement block, and the common structure extraction module. It extracts multi-scale context using dilated convolution and depthwise convolution with different dilation rates. It outputs the salient feature map at the current scale, where the fifth-level output is the final salient target image.

2. The common feature saliency target detection method based on lightweight networks according to claim 1, characterized in that: The first MobileNetV2 backbone network performs five layers of cascaded feature extraction on the input color visible light image. Each layer receives the output of the previous layer and generates a corresponding color feature map. The first to fifth layers output the first color visible light feature map, respectively. Second color visible light characteristic map Third color visible light characteristic map Fourth color visible light characteristic map Fifth color visible light characteristic map The second MobileNetV2 backbone network uses a dual-branch processing method to perform five-layer cascaded feature extraction on the input infrared image. Each layer receives the output of the previous layer and generates a corresponding infrared image feature map. The first to fifth layers output the first infrared image feature map respectively. Second infrared image feature map Third infrared image feature map Fourth infrared image feature map Fifth infrared image feature map Simultaneously, a five-layer concatenated feature extraction process is performed on the input infrared image. Each layer sequentially receives the output of the previous layer and generates a corresponding depth image feature map. The first to fifth layers respectively output the first depth image feature map. Second depth image feature map Third-depth image feature map Fourth depth image feature map Fifth depth image feature map .

3. The common feature saliency target detection method based on lightweight networks according to claim 1, characterized in that: The common structure extraction module includes five identical blocks for extracting geometric structure information common to infrared and depth images, enabling cross-modal complementary fusion. Each common structure extraction block consists of a first channel max pooling layer, a second channel max pooling layer, a first 1×1 convolutional layer, a first sigmoid activation layer, a first 3×3 convolutional layer, a first batch normalization (BN) layer, a second batch normalization (BN) layer, a first ReLU activation layer, a second ReLU activation layer, and a first 3×3 depth convolutional layer.

4. The common feature saliency target detection method based on lightweight networks according to claim 3, characterized in that: No. The implementation process of the shared structure extraction block is as follows: The first input terminal of the common structure extraction block receives Second input terminal receives Third input terminal receives ; The input terminal of the first channel maximum pooling layer receives The feature map output by the output terminal ; The input terminal of the second channel maximum pooling layer receives... The feature map output by the output terminal ;Will and Element-wise multiplication yields the feature map. ;Will and Element-wise multiplication yields the feature map. ;Will and By stitching along the channels, a feature map is obtained. The receiver of the first 1×1 convolutional layer receives... The receiver of the first sigmoid activation layer receives the feature map output by the output of the first 1×1 convolutional layer, and the output of the first sigmoid activation layer outputs the feature map. ;Will , and By stitching along the channels, a feature map is obtained. The input of the first 3×3 convolutional layer receives... The input of the first Batch Normalization (BN) layer receives the feature map output from the first 3×3 convolutional layer, and the input of the first ReLU activation layer receives the feature map output from the first BN layer. The output of the first ReLU activation layer also receives the feature map. ;Will and By stitching along the channels, a feature map is obtained. ;Will and Element-wise addition yields the feature map. The input of the first 3×3 depth convolutional layer receives... The input of the second BN layer receives the feature map output from the first 3×3 depth convolutional layer, and the input of the second ReLU activation layer receives the feature map output from the second BN layer. ;in, , ,and and One-to-one correspondence; for the first common structure extraction block, the third receiving end has no input, at this time and They are exactly the same, where DT represents both depth and infrared image modes.

5. The common feature saliency target detection method based on lightweight networks according to claim 1, characterized in that: The bimodal fusion module includes five identical bimodal fusion blocks for semantic information fusion strategies, addressing the high-quality fusion problem under trimodal conditions. Each bimodal fusion block consists of a first global average pooling layer, a second global average pooling layer, a second 1×1 convolutional layer, a third 1×1 convolutional layer, a fourth 1×1 convolutional layer, a fifth 1×1 convolutional layer, a sixth 1×1 convolutional layer, a seventh 1×1 convolutional layer, a third ReLU activation layer, a fourth ReLU activation layer, a fifth ReLU activation layer, a second sigmoid activation layer, a third sigmoid activation layer, a fourth sigmoid activation layer, a fifth sigmoid activation layer, a third channel max pooling layer, a fourth channel max pooling layer, a first channel average pooling layer, a second channel average pooling layer, a second 3×3 depthwise convolutional layer, and a third batch normalization (BN) layer.

6. The common feature saliency target detection method based on lightweight networks according to claim 5, characterized in that: No. The implementation process of the dual-modal fusion block is as follows: The first input of the dual-modal fusion block receives The second receiving end receives The third input terminal receives ;Will The feature map is obtained by sequentially passing through a first global average pooling layer, a second 1×1 convolutional layer, a third ReLU activation layer, a third 1×1 convolutional layer, and a second sigmoid activation layer. ;Will The feature map is obtained by sequentially passing through the second global average pooling layer, the fourth 1×1 convolutional layer, the fourth ReLU activation layer, the fifth 1×1 convolutional layer, and the third sigmoid activation layer. ;Will and Element-wise multiplication yields the feature map. ;Will and Element-wise addition yields the feature map. ;Will and Element-wise multiplication yields the feature map. ;Will and Element-wise addition yields the feature map. ;Will After passing through the third channel max pooling layer, the feature map is obtained. ;Will After passing through the first channel average pooling layer, the feature map is obtained. ;Will and By stitching along the channels, a feature map is obtained. ;Will The feature map is obtained by sequentially passing through the sixth 1×1 convolutional layer and the fourth sigmoid activation layer. ;Will and Element-wise multiplication yields the feature map. ;Will After passing through the fourth channel max pooling layer, the feature map is obtained. ;Will After passing through the second channel average pooling layer, the feature map is obtained. ;Will and By stitching along the channels, a feature map is obtained. ;Will The feature map is obtained by sequentially passing through the seventh 1×1 convolutional layer and the fifth sigmoid activation layer. ;Will and Element-wise multiplication yields the feature map. ;Will and By stitching along the channels, a feature map is obtained. ;Will and By stitching along the channels, a feature map is obtained. ;Will The feature map is obtained by sequentially passing the second 3×3 depth convolutional layer, the third BN layer, and the fifth ReLU activation layer. ;in, , ,and and One-to-one correspondence; for the first dual-mode fusion block, the third receiver has no input, at this time and Exactly the same.

7. The common feature saliency target detection method based on lightweight networks according to claim 1, characterized in that: The multi-scale feature refinement module includes five structurally identical feature refinement blocks used to efficiently transfer and fuse semantic and detailed information across different scales and levels. Each feature refinement block consists of an eighth 1×1 convolutional layer, a ninth 1×1 convolutional layer, a tenth 1×1 convolutional layer, an eleventh 1×1 convolutional layer, a twelfth 1×1 convolutional layer, a thirteenth 1×1 convolutional layer, a third 3×3 depth convolutional layer, a fourth 3×3 depth convolutional layer, a fifth 3×3 depth convolutional layer, a first dilated convolutional layer with a dilation rate of 3, a second dilated convolutional layer with a dilation rate of 5, a third dilated convolutional layer with a dilation rate of 7, a second 3×3 convolutional layer, a fourth BN layer, a sixth ReLU activation layer, a seventh ReLU activation layer, a first channel attention layer, and a first upsampling layer.

8. The common feature saliency target detection method based on lightweight networks according to claim 7, characterized in that: No. The implementation process of the feature refining block is as follows: The first input of the feature refining block receives The second input terminal receives The third input terminal receives ;Will , , Element-wise addition yields the feature map. ;Will The inputs are fed into the eighth, ninth, tenth, eleventh, and twelfth 1×1 convolutional layers, respectively, to obtain the corresponding feature maps. Feature map Feature map Feature map Feature map ;Will and Perform element-wise addition to obtain the feature map. ;Will The feature map is obtained by sequentially passing through a third 3×3 depth convolutional layer and a first dilated convolutional layer. ;Will and Perform element-wise addition to obtain the feature map. ;Will The feature map is obtained by sequentially passing through a fourth 3×3 depth convolutional layer and a second dilated convolutional layer. ;Will and Perform element-wise addition to obtain the feature map. ;Will The feature map is obtained by sequentially passing through a fifth 3×3 depth convolutional layer and a third dilated convolutional layer. ;Will , , , By stitching along the channels, a feature map is obtained. ;Will The feature map is obtained by sequentially passing through the second 3×3 convolutional layer, the fourth BN layer, the sixth ReLU activation layer, and the first channel attention layer. ;Will and Element-wise addition yields the feature map. ; The input of the seventh ReLU activation layer receives The feature map output from the seventh ReLU activation layer is also the feature map of the seventh ReLU activation layer. Feature map of the first output of the multi-scale feature extraction block. ;Will The feature map obtained after the thirteenth 1×1 convolutional layer and the first upsampling layer is also the... Feature map at the second output of the multi-scale feature extraction block ;in, , ,and and One-to-one correspondence; for the fifth feature refinement block, the third receiver has no input, at this time only for , When adding elements, where, As the final salient target image, RDT represents three modes: color visible light, depth, and infrared.

9. The common feature saliency target detection method based on lightweight networks according to claim 1, characterized in that: The process of obtaining the neural network training model is as follows: each pair of color visible light images and their corresponding infrared and depth images from the training set are input into the neural network for network training, and the loss function is calculated before the end of each round of network training. To optimize the neural network, a trained neural network model is obtained after a total of 190-210 rounds of network training: in, This represents the salient target image output by the multi-scale feature extraction module of the neural network. Represents a label image. This represents the weighted binary cross-entropy loss. This indicates the weighted average loss.

10. The common feature saliency target detection method based on lightweight networks according to claim 1, characterized in that: The process of using a neural network training model to predict the salient target image of the test image pair is as follows: First, arbitrarily select a pair of original color visible light images and their corresponding original infrared and original depth images; then, perform a downsampling operation on the pair of original color visible light images, original infrared images, and original depth images. The size of the downsampled image is... The test image pairs are then input into the trained neural network model to predict the corresponding salient target images.

Images