A common feature saliency target detection method based on lightweight networks

Abstract

This invention discloses a salient target detection method based on shared features using a lightweight network. It constructs a training set containing color visible light images, infrared images, and depth images, and builds a lightweight neural network. The lightweight neural network mainly consists of a shared structure extraction module, a dual-modal fusion module, and a multi-scale feature refinement module. Several pairs of color visible light images and their corresponding infrared and depth images from the training set are input into the neural network for several training iterations. After training, a neural network training model is obtained. The neural network training model is used to predict test image pairs to obtain salient target images, effectively improving the accuracy of salient target detection with low parameter and computational requirements.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of salient object detection, and particularly relates to a common feature salient object detection method based on a lightweight network. BACKGROUND

[0002] Salient object detection aims to capture and segment the most attractive regions or objects in images or videos that attract human attention. As an important preprocessing step, salient object detection has been widely used in computer vision and image processing tasks, such as image segmentation, object tracking, image retrieval, and image quality assessment. In recent years, convolutional neural networks and deep learning have pushed the performance of salient object detection to a new height due to their strong learning ability and excellent performance in feature extraction. However, in some extreme scenarios (such as low light and chaotic scenes), it is often difficult to extract valuable information from color visible light images, which affects the effectiveness of salient object detection models in complex conditions. Unlike color visible light images, thermal infrared cameras can capture thermal information that is less affected by environmental factors such as darkness and bad weather, thus reflecting temperature differences, boundaries, and geometric shapes. Therefore, deploying thermal infrared devices to collect thermal information and using color visible light and infrared image salient object detection models to record objects can improve the perception ability of the model in more scenarios. Depth images contain geometric structure information of objects, and spatial information of different objects can better help the model perceive object information in complex color scenes. Therefore, in recent years, more and more attention has been paid to the research on salient object detection of color visible light images combined with infrared images and depth images.

[0003] On the other hand, existing visible light-depth-infrared (RGB-D-T) salient object detection models have made significant breakthroughs in precision, but these high-precision models usually require high computational cost and large model size to process multi-modal information, which makes them only deployable in graphics processing unit facilities with high performance requirements but limited computing power, and cannot meet the needs of some actual scenarios in terms of computing latency and bandwidth cost, such as autonomous driving, monitoring Internet of Things, etc.

[0004] Currently, there are some lightweight RGB-D-T salient object detection models, which are mainly based on convolutional neural network (CNN). These methods usually use a lightweight backbone network to extract features from three modalities, and then perform multi-modal and multi-level feature fusion through various lightweight modules designed with care. Although these methods successfully compress the model size and can be deployed on some resource-constrained devices, there is still room for improvement in model volume, and the accuracy cannot meet the needs. Therefore, how to enhance the prediction performance of lightweight RGB-D-T salient object detection model while keeping low model size has become a problem to be solved. SUMMARY

[0005] The present application aims at the above-mentioned deficiencies, and proposes a common feature salient object detection method based on lightweight network, which can effectively improve the accuracy of salient object detection with low parameter amount and computation amount.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical scheme: a common feature salient object detection method based on lightweight network, comprising the following steps: first, constructing a three-modal training set containing multiple pairs of registered color visible light images, infrared images and depth images, and building a lightweight neural network; then inputting the three-modal images in the training set into the lightweight neural network for multi-round network training, obtaining a neural network training model after network training is completed; and again using the neural network training model to predict the test image pair, and obtaining the salient object image of the test image pair, the test image pair including a color visible light image and corresponding infrared image and depth image;

[0007] The lightweight neural network includes a feature extraction module, a common structure extraction module, a dual-modal fusion module and a multi-scale feature refining module connected in sequence;

[0008] The feature extraction module includes a first MobileNetV2 backbone network for extracting feature information and scale information of the color visible light image, and a second MobileNetV2 backbone network for extracting feature information and scale information of the infrared image and the depth image;

[0009] The common structure extraction block is used to perform channel maximum pooling on the input depth feature map and infrared feature map, and then generate common structure weights through element multiplication and splicing operation, and output geometrically aligned common structure feature maps through depth convolution and residual connection;

[0010] The dual-modal fusion block is used to perform global average pooling on the color visible light feature and the common structure feature, and then generate modal attention weights through a fully connected layer; the dual-modal features are weighted and fused and outputted;

[0011] The feature refining block is used for receiving features from the bimodal fusion module, the upper-level refining block and the common structure extraction module, and adopting different inflation rates of the empty convolution and the deep convolution to extract multi-scale contexts; and outputting a saliency feature map of a current scale, wherein the fifth level output is a final saliency target image.

[0012] As an improvement, the first MobileNetV2 backbone network performs five-layer serial feature extraction on the input color visible light image, each layer sequentially receives the output of the previous layer and generates a corresponding color feature map, and the first to fifth layers output the first color visible light feature map , the second color visible light feature map , the third color visible light feature map , the fourth color visible light feature map , and the fifth color visible light feature map ; the second MobileNetV2 backbone network double-branch processing, five-layer serial feature extraction is performed on the input infrared image, each layer sequentially receives the output of the previous layer and generates a corresponding infrared image feature map, and the first to fifth layers output the first infrared image feature map , the second infrared image feature map , the third infrared image feature map , the fourth infrared image feature map , and the fifth infrared image feature map , and five-layer serial feature extraction is performed on the input infrared image, each layer sequentially receives the output of the previous layer and generates a corresponding depth image feature map, and the first to fifth layers output the first depth image feature map , the second depth image feature map , the third depth image feature map , the fourth depth image feature map , and the fifth depth image feature map .

[0013] As an improvement, the common structure extraction module includes five structure same blocks for refining the common geometric structure information in the infrared image and the depth image, realizing the common structure extraction block of cross-modal complementary fusion, and the common structure extraction block is composed of a first channel maximum pooling layer, a second channel maximum pooling layer, a first 1×1 convolution layer, a first Sigmoid activation layer, a first 3×3 convolution layer, a first BN layer, a second BN layer, a first ReLU activation layer, a second ReLU activation layer, and a first 3×3 depth convolution layer.

[0014] As an improvement, the first common structure extraction block includes a first input end, a second input end, a first 1×1 convolution layer, a first Sigmoid activation layer, a first 3×3 convolution layer, a first BN layer, a second BN layer, a first ReLU activation layer, a second ReLU activation layer, and a first 3×3 depth convolution layer. The first input end of the common structure extraction block receives , the third input end receives ; the input end of the first channel max pooling layer receives , the output end outputs a feature map ; the input end of the second channel max pooling layer receives , the output end outputs a feature map ; the and the are element-wise multiplied to obtain a feature map ; the and the are element-wise multiplied to obtain a feature map ; the and the are spliced along the channel to obtain a feature map ; the receiving end of the first 1x1 convolutional layer receives , the receiving end of the first Sigmoid activation layer receives the feature map output by the output end of the first 1x1 convolutional layer, and the output end of the first Sigmoid activation layer outputs a feature map ; the , and the are spliced along the channel to obtain a feature map ; the input end of the first 3x3 convolutional layer receives , the input end of the first BN layer receives the feature map output by the output end of the first 3x3 convolutional layer, the input end of the first ReLU activation layer receives the feature map output by the output end of the first BN layer, and the output end of the first ReLU activation layer outputs a feature map ; the and the are spliced along the channel to obtain a feature map ; the and the are element-wise added to obtain a feature map ; the input end of the first 3x3 depth convolutional layer receives , the input end of the second BN layer receives the feature map output by the output end of the first 3x3 depth convolutional layer, the input end of the second ReLU activation layer receives the feature map output by the output end of the second BN layer, and the output end of the second ReLU activation layer outputs a feature map ; wherein , , and and are one-to-one correspondence; for the first common structure extraction block, the third receiving end has no input, at this time and are exactly the same, wherein DT represents depth and infrared image two modalities.

[0015] As an improvement, the bimodal fusion module includes five structures same for the fusion strategy for semantic information, solves the bimodal fusion block of the high-quality fusion problem under three modes, and the bimodal fusion block is composed of a first global average pooling layer, a second global average pooling layer, a second 1*1 convolution layer, a third 1*1 convolution layer, a fourth 1*1 convolution layer, a fifth 1*1 convolution layer, a sixth 1*1 convolution layer, a seventh 1*1 convolution layer, a third ReLU activation layer, a fourth ReLU activation layer, a fifth ReLU activation layer, a second sigmoid activation layer, a third sigmoid activation layer, a fourth sigmoid activation layer, a fifth sigmoid activation layer, a third channel maximum pooling layer, a fourth channel maximum pooling layer, a first channel average pooling layer, a second channel average pooling layer, a second 3*3 deep convolution layer, and a third BN layer.

[0016] As an improvement, the first The implementation process of the bimodal fusion block is as follows: the first input end of the bimodal fusion block receives The second receiving end receives The third input end receives ; and is sequentially subjected to the first global average pooling layer, the second 1*1 convolution layer, the third ReLU activation layer, the third 1*1 convolution layer, and the second sigmoid activation layer to obtain a feature map ; and is sequentially subjected to the second global average pooling layer, the fourth 1*1 convolution layer, the fourth ReLU activation layer, the fifth 1*1 convolution layer, and the third sigmoid activation layer to obtain a feature map ; and is subjected to element multiplication with to obtain a feature map ; and is subjected to element addition with to obtain a feature map ; and is subjected to element multiplication with to obtain a feature map ; and is subjected to element addition with to obtain a feature map ; and is subjected to the third channel maximum pooling layer to obtain a feature map ; and is subjected to the first channel average pooling layer to obtain a feature map ; and is spliced along the channel with to obtain a feature map ; and is subjected to the second 3*3 deep convolution layer and the third BN layer to obtain a feature map ; andThe feature map is obtained by sequentially passing the sixth 1x1 convolutional layer, the fourth sigmoid activation layer ; the and are multiplied element by element to obtain a feature map ; the passes through the fourth channel maximum pooling layer to obtain a feature map ; the passes through the second channel average pooling layer to obtain a feature map ; the and are spliced along the channel to obtain a feature map ; the passes through the seventh 1x1 convolutional layer, the fifth sigmoid activation layer in sequence to obtain a feature map ; the and are multiplied element by element to obtain a feature map ; the and are spliced along the channel to obtain a feature map ; the and are spliced along the channel to obtain a feature map ; the passes through the second 3x3 deep convolutional layer, the third BN layer, the fifth ReLU activation layer in sequence to obtain a feature map ; wherein, , , and correspond to one by one; for the first dual-modal fusion block, the third receiving end has no input, at this time is exactly the same as .

[0017] As an improvement, the multi-scale feature refining module includes five feature refining blocks with the same structure for efficiently transmitting and fusing semantic and detail information between different scales and levels. The feature refining block is composed of an eighth 1x1 convolutional layer, a ninth 1x1 convolutional layer, a tenth 1x1 convolutional layer, an eleventh 1x1 convolutional layer, a twelfth 1x1 convolutional layer, a thirteenth 1x1 convolutional layer, a third 3x3 deep convolutional layer, a fourth 3x3 deep convolutional layer, a fifth 3x3 deep convolutional layer, a first dilated convolutional layer with an expansion rate of 3, a second dilated convolutional layer with an expansion rate of 5, a third dilated convolutional layer with an expansion rate of 7, a second 3x3 convolutional layer, a fourth BN layer, a sixth ReLU activation layer, a seventh ReLU activation layer, a first channel attention layer, and a first up-sampling layer.

[0018] As an improvement, the first feature refining block is implemented as follows: the The first input of the feature refining block receives The second input terminal receives The third input terminal receives ;Will , , Element-wise addition yields the feature map. ;Will The inputs are fed into the eighth, ninth, tenth, eleventh, and twelfth 1×1 convolutional layers, respectively, to obtain the corresponding feature maps. Feature map Feature map Feature map Feature map ;Will and Perform element-wise addition to obtain the feature map. ;Will The feature map is obtained by sequentially passing through a third 3×3 depth convolutional layer and a first dilated convolutional layer. ;Will and Perform element-wise addition to obtain the feature map. ;Will The feature map is obtained by sequentially passing through a fourth 3×3 depth convolutional layer and a second dilated convolutional layer. ;Will and Perform element-wise addition to obtain the feature map. ;Will The feature map is obtained by sequentially passing through a fifth 3×3 depth convolutional layer and a third dilated convolutional layer. ;Will , , , By stitching along the channels, a feature map is obtained. ;Will The feature map is obtained by sequentially passing through the second 3×3 convolutional layer, the fourth BN layer, the sixth ReLU activation layer, and the first channel attention layer. ;Will and Element-wise addition yields the feature map. The input of the seventh ReLU activation layer receives... The feature map output from the seventh ReLU activation layer is also the feature map of the seventh ReLU activation layer. Feature map of the first output of the multi-scale feature extraction block. ;Will The feature map obtained after the thirteenth 1×1 convolutional layer and the first upsampling layer is also the... Feature map at the second output of the multi-scale feature extraction block ;in, , ,and and One-to-one correspondence; for the fifth feature refinement block, the third receiver has no input, at this time only for , When adding elements, where, As the final salient target image, RDT represents three modes: color visible light, depth, and infrared.

[0019] As an improvement, the process of obtaining the neural network training model is as follows: each pair of color visible light images and their corresponding infrared and depth images from the training set are input into the neural network for network training, and the loss function is calculated before the end of each round of network training. To optimize the neural network, a trained neural network model is obtained after a total of 190-210 rounds of network training:

[0020]

[0021] in, This represents the salient target image output by the multi-scale feature extraction module of the neural network. Represents a label image. This represents the weighted binary cross-entropy loss. This indicates the weighted average loss.

[0022] As an improvement, a neural network-trained model is used to predict the salient target image of the test image pair. The process is as follows: First, arbitrarily select a pair of original color visible light images and their corresponding original infrared and depth images; then, downsample the pair of original color visible light images, original infrared images, and original depth images. The size of the downsampled image is... The test image pairs are then input into the trained neural network model to predict the corresponding salient target images.

[0023] Compared with the prior art, the advantages of the present invention are as follows:

[0024] The method of the present application significantly improves the perception ability in complex environments (such as low illumination, chaotic scenes) by fusing three modal information of color visible light images, infrared images and depth images. The infrared image can capture thermal information that is not affected by environmental factors such as darkness and bad weather, and the depth image provides geometric structure information of the object, which helps to more accurately perceive the object in a complex color scene. The method of the present application significantly improves the perception ability in complex environments (such as low illumination, chaotic scenes) by fusing three modal information of color visible light images, infrared images and depth images. The infrared image can capture thermal information that is not affected by environmental factors such as darkness and bad weather, and the depth image provides geometric structure information of the object, which helps to more accurately perceive the object in a complex color scene.

[0025] The method of the present application adopts a lightweight neural network, including a lightweight backbone network (such as MobileNetV2) and a carefully designed lightweight module (such as a common structure extraction module, a dual-modal fusion module and a multi-scale feature refining module). Based on the common structural features in infrared and depth images, the same backbone network is used for structural feature extraction and optimization, simplifying the expression of modal, reducing the number of backbone networks required for modal feature extraction, achieving the purpose of reducing model parameter quantity and improving model running speed, so that the model has smaller model size while maintaining high performance, reducing the calculation cost, helping to deploy the model on resource-limited devices, and expanding the application range of salient object detection technology. And the present application method designs a dual-modal fusion module for semantic information, and a multi-scale feature refining module in the process of three modal interaction and fusion, which can efficiently transmit and fuse semantic and detail information between different scales and levels, and is assisted by saliency supervision strategy. This combination not only strengthens the characteristics of each modal, but also deeply excavates the shared information, significantly improving the comprehensive performance of multi-modal salient object detection. At the same time, the common structure extraction module refines and aligns the common geometric structure information between the depth image and the infrared image; this module fully utilizes the advantages of two modalities that are not sensitive to light and consistent in geometry, combined with the texture semantics of the RGB branch, to realize cross-modal complementary fusion, thereby effectively improving the accuracy and robustness of salient object detection.

[0026] The method of the present application adopts specific loss functions (such as weighted binary cross-entropy loss and weighted intersection over union loss) for network training, which helps to optimize the performance of the neural network and improve the accuracy of salient object detection. By constructing a three-modal training set containing multiple pairs of registration, the present application method can make more full use of training data and improve the generalization ability of the model. By combining multi-modal information and optimizing the network structure, the present application method can realize high-precision salient object detection. This helps to improve the performance and user experience of related applications. BRIEF DESCRIPTION OF DRAWINGS

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0028] Figure 1 This is a diagram illustrating the overall implementation framework of the method of the present invention;

[0029] Figure 2 A schematic diagram illustrating the structural composition of the neural network constructed using the method of this invention;

[0030] Figure 3 A schematic diagram of the composition structure of the common structure extraction block in the neural network constructed by the method of the present invention;

[0031] Figure 4 A schematic diagram illustrating the structural composition of the dual-modal fusion block in the neural network constructed by the method of this invention;

[0032] Figure 5 This is a schematic diagram of the composition structure of the feature refinement block in the neural network constructed by the method of the present invention. Detailed Implementation

[0033] like Figures 1 to 5 As shown, a common feature saliency target detection method based on a lightweight network includes the following steps: First, a three-modal training set containing multiple pairs of registered color visible light images, infrared images, and depth images is constructed, and a lightweight neural network is built; then, the three-modal images in the training set are input into the lightweight neural network for multiple rounds of network training, and a neural network training model is obtained after the network training is completed; and the neural network training model is used again to predict the test image pairs to obtain the saliency target images of the test image pairs, where the test image pairs include a color visible light image and its corresponding infrared image and depth image;

[0034] The training set is constructed as follows: at least 1000 pairs of original color visible light images and their corresponding original infrared and depth images are selected; then, each original color visible light image and its corresponding original infrared and depth images are downsampled, and the size of the downsampled image is [size missing]. Then all sizes are The training set consists of color visible light images and their corresponding infrared and depth images; among them, 1048 pairs of original color visible light images and their corresponding original infrared and depth images are selected. .

[0035] The lightweight neural network consists of a feature extraction module, a common structure extraction module, a dual-modal fusion module, and a multi-scale feature refinement module connected in sequence.

[0036] The feature extraction module comprises a first MobileNetV2 backbone network for extracting feature information and scale information of the color visible light image, and a second MobileNetV2 backbone network for extracting feature information and scale information of the infrared image and the depth image;

[0037] The first layer of the first MobileNetV2 backbone network performs feature extraction on the color visible light image and outputs a first color visible light feature map The second layer of the first MobileNetV2 backbone network receives and outputs a second color visible light feature map The third layer of the first MobileNetV2 backbone network receives and outputs a third color visible light feature map The third layer, the fourth layer of the first MobileNetV2 backbone network receives and outputs a fourth color visible light feature map The fifth layer of the first MobileNetV2 backbone network receives and outputs a fifth color visible light feature map The first layer of the second MobileNetV2 backbone network performs feature extraction on the infrared image and outputs a first infrared image feature map The second layer of the second MobileNetV2 backbone network receives and outputs a second infrared image feature map The third layer of the second MobileNetV2 backbone network receives and outputs a third infrared image feature map The fourth layer of the second MobileNetV2 backbone network receives and outputs a fourth infrared image feature map The fifth layer of the second MobileNetV2 backbone network receives and outputs a fifth infrared image feature map The first layer of the second MobileNetV2 backbone network performs feature extraction on the depth image and outputs a first depth image feature map The second layer of the second MobileNetV2 backbone network receives and outputs a second depth image feature map The third layer of the second MobileNetV2 backbone network receives and outputs a third depth image feature map The fourth layer of the second MobileNetV2 backbone network receives and outputs a fourth depth image feature map , the fifth layer of the second MobileNetV2 backbone network receives and outputs the fifth deep image feature map ;

[0038] The common structure extraction module includes five structure same blocks for extracting common geometric structure information in the infrared image and the depth image, and realizing cross-modal complementary fusion. The first input end of the first common structure extraction block receives , the second input end receives , and the third receiving end has no input. The feature map output by the output end of the first common structure extraction block is denoted as ; the first input end of the second common structure extraction block receives , the second input end receives , and the third input end receives . The feature map output by the output end of the second common structure extraction block is denoted as ; the first input end of the third common structure extraction block receives , the second input end receives , and the third input end receives . The feature map output by the output end of the third common structure extraction block is denoted as ; the first input end of the fourth common structure extraction block receives , the second input end receives , and the third input end receives . The feature map output by the output end of the fourth common structure extraction block is denoted as ; the first input end of the fifth common structure extraction block receives , the second input end receives , and the third input end receives . The feature map output by the output end of the fifth common structure extraction block is denoted as ; wherein, DT represents two modalities of the depth and infrared images.

[0039] The dual-modality fusion module includes five structure same blocks for solving the problem of high-quality fusion under three modalities, and the dual-modality fusion blocks are different in input and output. The first input end of the first dual-modality fusion block receives , the second input end receives , and the third receiving end has no input. The feature map output by the output end of the first dual-modality fusion block is denoted as ; the first input end of the second dual-modality fusion block receives , the second input end receives , and the third input end receives . The feature map output by the output end of the second dual-modality fusion block is denoted as ; the first input end of the third dual-modality fusion block receives , the second input end receives , the third input end receives , the output end of the third dual-modal fusion block outputs feature maps recorded as ; the first input end of the fourth dual-modal fusion block receives , the second input end receives , the third input end receives , the output end of the fourth dual-modal fusion block outputs feature maps recorded as ; the first input end of the fifth dual-modal fusion block receives , the second input end receives , the third input end receives , the output end of the fifth dual-modal fusion block outputs feature maps recorded as ; wherein RT represents two modalities of color visible light and infrared images;

[0040] The multi-scale feature refining module includes five feature refining blocks of the same structure but different inputs and outputs for efficiently transmitting and fusing semantic and detail information between different scales and levels. The first input end of the first feature refining block receives , the second input end receives , the third input end receives , the first output end of the first feature refining block outputs feature maps recorded as , the second output end outputs feature maps recorded as ; the first input end of the second feature refining block receives , the second input end receives , the third input end receives , the first output end of the second feature refining block outputs feature maps recorded as , the second output end outputs feature maps recorded as ; the first input end of the third feature refining block receives , the second input end receives , the third input end receives , the first output end of the third feature refining block outputs feature maps recorded as , the second output end outputs feature maps recorded as ; the first input end of the fourth feature refining block receives , the second input end receives , the third input end receives , the first output end of the fourth feature refining block outputs feature maps recorded as , the second output end outputs feature maps recorded as ; the first input end of the fifth feature refining block receives , the second input end receives , the third input end has no input, the feature map outputted by the first output end of the fifth feature refining block is recorded as , the feature map outputted by the second output end is recorded as , wherein, As the final saliency target image, RDT represents three modalities of color visible light, depth and infrared;

[0041] Here, the size of the color visible light image, the infrared image and the depth image is In this embodiment, Take 320*320, , and The size of each of , , and is , , and is , , and is , , and is , , and is , , and is , , and is , , and is , , and is , , , , and is ;

[0042] The common structure extraction block is used for performing channel max pooling on the input depth feature map and infrared feature map; common structure weight is generated through element multiplication and splicing operation; a geometrically aligned common structure feature map is output by using depth convolution and residual connection; the common structure extraction block is composed of a first channel max pooling layer, a second channel max pooling layer, a first 1*1 convolution layer, a first Sigmoid activation layer, a first 3*3 convolution layer, a first BN layer, a second BN layer, a first ReLU activation layer, a second ReLU activation layer and a first 3*3 depth convolution layer;

[0043] The first channel max pooling layer is used for performing channel max pooling on the input depth feature map and infrared feature map; the common structure weight is generated through element multiplication and splicing operation; the geometrically aligned common structure feature map is output by using depth convolution and residual connection; the common structure extraction block is composed of a first channel max pooling layer, a second channel max pooling layer, a first 1*1 convolution layer, a first Sigmoid activation layer, a first 3*3 convolution layer, a first BN layer, a second BN layer, a first ReLU activation layer, a second ReLU activation layer and a first 3*3 depth convolution layer; The first channel max pooling layer is used for performing channel max pooling on the input depth feature map and infrared feature map; the common structure weight is generated through element multiplication and splicing operation; the geometrically aligned common structure feature map is output by using depth convolution and residual connection; the common structure extraction block is composed of a first channel max pooling layer, a second channel max pooling layer, a first 1*1 convolution layer, a first Sigmoid activation layer, a first 3*3 convolution layer, a first BN layer, a second BN layer, a first ReLU activation layer, a second ReLU activation layer and a first 3*3 depth convolution layer; The first input end of the common structure extraction block receives , the second input end receives , and the third input end receives ; the input end of the first channel max pooling layer receives , and the output end outputs a feature map denoted as ; the input end of the second channel max pooling layer receives , and the output end outputs a feature map denoted as ; element multiplication is performed on and to obtain a feature map denoted as ; element multiplication is performed on and to obtain a feature map denoted as ; splicing is performed on and along the channel to obtain a feature map denoted as ; the receiving end of the first 1*1 convolution layer receives , the receiving end of the first Sigmoid activation layer receives the feature map output by the output end of the first 1*1 convolution layer, and the output end of the first Sigmoid activation layer outputs a feature map denoted as ; splicing is performed on , and along the channel to obtain a feature map denoted as ; the input end of the first 3*3 convolution layer receives , the input end of the first BN layer receives the feature map output by the output end of the first 3*3 convolution layer, the input end of the first ReLU activation layer receives the feature map output by the output end of the first BN layer, and the output end of the first ReLU activation layer outputs a feature map denoted as ; splicing is performed on and along the channel to obtain a feature map denoted as ; element addition is performed on and to obtain a feature map The input of the first 3×3 depth convolutional layer receives... The input of the second BN layer receives the feature map output from the first 3×3 depth convolutional layer, and the input of the second ReLU activation layer receives the feature map output from the second BN layer. The feature map output from the second ReLU activation layer is denoted as... ;in, , ,and and One-to-one correspondence; for the first common structure extraction block, the third receiving end has no input, at this time and They are exactly the same; here, element-wise multiplication, concatenation along channels, and element-wise addition are all common operations in neural networks; channel max pooling layers are common modules in neural networks;

[0044] The bimodal fusion block is used to perform global average pooling on color visible light features and common structural features; generate modal attention weights through fully connected layers; and weightedly fuse bimodal features and output them. The bimodal fusion block consists of a first global average pooling layer, a second global average pooling layer, a second 1×1 convolutional layer, a third 1×1 convolutional layer, a fourth 1×1 convolutional layer, a fifth 1×1 convolutional layer, a sixth 1×1 convolutional layer, a seventh 1×1 convolutional layer, a third ReLU activation layer, a fourth ReLU activation layer, a fifth ReLU activation layer, a second sigmoid activation layer, a third sigmoid activation layer, a fourth sigmoid activation layer, a fifth sigmoid activation layer, a third channel max pooling layer, a fourth channel max pooling layer, a first channel average pooling layer, a second channel average pooling layer, a second 3×3 depthwise convolutional layer, and a third batch normalization (BN) layer.

[0045] No. The implementation process of the dual-modal fusion block is as follows: The first input of the dual-modal fusion block receives The second receiving end receives The third input terminal receives The receiver of the first global average pooling layer receives... ,second The receiver of the convolutional layer receives the feature map output from the first global average pooling layer; the receiver of the third ReLU activation layer receives the feature map output from the second 1×1 convolutional layer; the receiver of the third 1×1 convolutional layer receives the feature map output from the third ReLU activation layer; the receiver of the second sigmoid activation layer receives the feature map output from the third 1×1 convolutional layer; and the feature map output from the second sigmoid activation layer is denoted as... The receiver of the second global average pooling layer receives... ,fourth The receiver of the convolutional layer receives the feature map output from the second global average pooling layer; the receiver of the fourth ReLU activation layer receives the feature map output from the fourth 1×1 convolutional layer; the receiver of the fifth 1×1 convolutional layer receives the feature map output from the fourth ReLU activation layer; and the receiver of the third sigmoid activation layer receives the feature map output from the fifth 1×1 convolutional layer. The feature map output from the third sigmoid activation layer is denoted as... ;Will and Element-wise multiplication yields a feature map, denoted as . ;Will and Element-wise addition yields the feature map. ;Will and Element-wise multiplication yields a feature map, denoted as . ;Will and Element-wise addition yields the feature map. The receiver of the third channel's maximum pooling layer receives... The feature map output from the third channel max pooling layer is denoted as... The receiver of the first channel average pooling layer receives... The feature map output from the first channel average pooling layer is denoted as... ;Will and The feature map is obtained by splicing along the channel. The receiver of the sixth 1×1 convolutional layer receives... The receiver of the fourth sigmoid activation layer receives the feature map output from the output of the sixth 1×1 convolutional layer. The feature map output from the fourth sigmoid activation layer is denoted as... ;Will and Element-wise multiplication yields a feature map, denoted as . The receiver of the fourth channel's maximum pooling layer receives... The feature map output from the fourth channel max pooling layer is denoted as... The receiver of the second channel average pooling layer receives... The feature map output from the first channel average pooling layer is denoted as... ;Will and The feature map is obtained by splicing along the channel. The receiver of the seventh 1×1 convolutional layer receives... The receiver of the fifth sigmoid activation layer receives the feature map output from the output of the seventh 1×1 convolutional layer. The feature map output from the fifth sigmoid activation layer is denoted as... ;Will and Element-wise multiplication yields a feature map, denoted as . ;Will and The feature map is obtained by splicing along the channel. ;Will and The feature map is obtained by splicing along the channel. The receiver of the second 3×3 depth convolutional layer receives... The receiving end of the third BN layer receives the feature map output from the output end of the second 3×3 depth convolutional layer, and the fifth ReLU activation layer receives the feature map output from the output end of the third BN layer. The feature map output from the output end of the fifth ReLU activation layer is denoted as... ;in, , ,and and One-to-one correspondence; for the first dual-mode fusion block, the third receiver has no input, at this time and They are exactly the same; here, global average pooling layer and channel average pooling layer are common modules in neural networks.

[0046] The feature refinement block receives features from the dual-modal fusion module, the previous refinement block, and the common structure extraction module; it extracts multi-scale context using dilated convolutions and depthwise convolutions with dilation rates of 3, 5, and 7; and outputs the salient feature map at the current scale, with the fifth-level output being the final salient target image.

[0047] The feature refinement block consists of the eighth 1×1 convolutional layer, the ninth 1×1 convolutional layer, the tenth 1×1 convolutional layer, the eleventh 1×1 convolutional layer, the twelfth 1×1 convolutional layer, the thirteenth 1×1 convolutional layer, the third 3×3 depth convolutional layer, the fourth 3×3 depth convolutional layer, the fifth 3×3 depth convolutional layer, the first dilated convolutional layer with an inflation rate of 3, the second dilated convolutional layer with an inflation rate of 5, the third dilated convolutional layer with an inflation rate of 7, the second 3×3 convolutional layer, the fourth BN layer, the sixth ReLU activation layer, the seventh ReLU activation layer, the first channel attention layer, and the first upsampling layer.

[0048] No. The implementation process of the feature refining block is as follows: the first input terminal of the feature refining block receives... The second input terminal receives The third input terminal receives ;Will , , Element-wise addition yields the feature map. ;Will The inputs are fed into the eighth, ninth, tenth, eleventh, and twelfth 1×1 convolutional layers, respectively, to obtain the corresponding feature maps. Feature map Feature map Feature map Feature map ;Will and Perform element-wise addition to obtain the feature map. The input of the third 3×3 depth convolutional layer receives... The input of the first dilated convolutional layer with an inflation rate of 3 receives the feature map output from the output of the third 3×3 depth convolutional layer. The feature map output from the first dilated convolutional layer with an inflation rate of 3 is denoted as... ;Will and Perform element-wise addition to obtain the feature map. The input of the fourth 3×3 depth convolutional layer receives... The input of the second dilated convolutional layer with a dilation rate of 5 receives the feature map output from the output of the fourth 3×3 depth convolutional layer. The feature map output from the second dilated convolutional layer with a dilation rate of 5 is denoted as... ;Will and Perform element-wise addition to obtain the feature map. The input of the fifth 3×3 depth convolutional layer receives... The input of the third dilated convolutional layer with an inflation rate of 7 receives the feature map output from the output of the fifth 3×3 depth convolutional layer. The feature map output from the third dilated convolutional layer with an inflation rate of 7 is denoted as... ;Will , , , The feature map is obtained by splicing along the channel. The input of the second 3×3 convolutional layer receives... The input of the fourth BN layer receives the feature map output from the second 3×3 convolutional layer; the input of the sixth ReLU activation layer receives the feature map output from the fifth BN layer; and the input of the first channel attention layer receives the feature map output from the sixth ReLU activation layer. The feature map output from the first channel attention layer is denoted as... ;Will and Adding elements together yields a feature map, denoted as . The input of the seventh ReLU activation layer receives... The feature map output from the seventh ReLU activation layer is also the feature map of the seventh ReLU activation layer. The feature map at the first output of the multi-scale feature extraction block is denoted as... The input of the thirteenth 1×1 convolutional layer receives... The input of the first upsampling layer receives the feature map output from the output of the thirteenth 1×1 convolutional layer. The feature map output from the first upsampling layer is also the feature map of the thirteenth 1×1 convolutional layer. The feature map at the second output of the multi-scale feature extraction block is denoted as... ;in, , ,and and One-to-one correspondence; for the fifth feature refinement block, the third receiver has no input, at this time only for , Add elements together;

[0049] The process of obtaining the neural network training model is as follows: each pair of color visible light images and their corresponding infrared and depth images from the training set are input into the neural network for network training. The loss function is calculated before the end of each round of network training. To optimize the neural network, a trained neural network model was obtained after a total of 200 rounds of network training:

[0050]

[0051] in, This represents the salient target image output by the multi-scale feature extraction module of the neural network. Represents a label image. This represents the weighted binary cross-entropy loss. Indicates the weighted average loss;

[0052] The process of using a neural network-trained model to predict the salient target image of a test image pair is as follows: First, arbitrarily select a pair of original color visible light images and their corresponding original infrared and depth images; then, downsample the pair of original color visible light images, original infrared images, and original depth images. The size of the downsampled image is... The test image pairs are then input into the trained neural network model to predict the corresponding salient target images.

[0053] To further demonstrate the feasibility and effectiveness of the method of the present invention, experiments were conducted on the method of the present invention.

[0054] In the experiment, the method of this invention was used to test a dataset of salient objects in color visible light images, depth images, and infrared images (VDT2048 dataset). The VDT2048 dataset contains a training set and a test set. The training set includes 1048 pairs of color visible light images, infrared images, and depth images, i.e., 1048 training image pairs. The test set includes 1000 pairs of color visible light images, infrared images, and depth images, i.e., 1000 test image pairs.

[0055] In this embodiment, four commonly used objective parameters and three commonly used objective parameters for measuring model lightweightness are selected to evaluate the performance of the method of the present invention. These are S-measure, E-measure, F-measure, Mean Absolute Error (MAE), Params (M), Flops (G), and FPS. Table 1 shows the correlation between salient target images and label images obtained by the method of the present invention on the VDT2048 dataset, as well as the lightweightness of the model.

[0056] Table 1 shows the S-measure, E-measure, F-measure, and Mean Absolute Error (MAE) between salient target images and label images obtained using the method of this invention on the VDT2048 dataset.

[0057]

[0058] The results shown in Table 1 reveal that the method of this invention achieves high S-measure, E-measure, and F-measure, and low MAE on existing trimodal salient object detection datasets, while also exhibiting low parameter and computational complexity and high computational speed. This indicates that the salient object images and label images obtained by the method of this invention are quite similar, and the method can effectively complete the trimodal salient object detection task with low parameter and computational complexity.

[0059] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0060] The units described in some embodiments of this disclosure can be implemented in software or in hardware. The described units can also be located in a processor, and the functions described above can be performed at least in part by one or more hardware logic components.

[0061] The above description only illustrates the preferred embodiments of the present invention and should not be construed as limiting the scope of the claims. The present invention is not limited to the above embodiments, and variations in its specific structure are permitted. All modifications made within the scope of the independent claims of this invention are also within the scope of protection of this invention.

Claims

1. A method for detecting salient object based on common features of lightweight network, characterized in that, The method comprises the following steps: First, a three-modal training set comprising multiple pairs of registered color visible light images, infrared images and depth images is constructed, and a lightweight neural network is built; then the three-modal images in the training set are input into the lightweight neural network for multiple rounds of network training, and a neural network training model is obtained after the network training is completed; and the neural network training model is used again to predict a test image pair, and a saliency target image of the test image pair is obtained, wherein the test image pair comprises a color visible light image and corresponding infrared and depth images; The lightweight neural network comprises a feature extraction module, a common structure extraction module, a dual-modal fusion module and a multi-scale feature refining module connected in sequence; The feature extraction module comprises a first MobileNetV2 backbone network for extracting feature information and scale information of a color visible light image, and a second MobileNetV2 backbone network for extracting feature information and scale information of an infrared image and a depth image; The common structure extraction block is used to perform channel max-pooling on the input depth feature map and infrared feature map, and then generate common structure weights through element multiplication and concatenation operation, and output geometrically aligned common structure feature maps through depth convolution and residual connection; The dual-modal fusion module is used to perform global average pooling on the color visible light feature and the common structure feature, and then generate modal attention weights through a fully connected layer; The dual-modal features are weighted and fused, and output; The multi-scale feature refining module is used to receive features from the dual-modal fusion module, the previous level refining block and the common structure extraction module, and extract multi-scale contexts through a depth convolution and a depth convolution with different dilation rates; and output a saliency feature map of the current scale, wherein the fifth level output is the final saliency target image. The obtaining process of the neural network training model is: inputting each pair of color visible light image and corresponding infrared image and depth image in the training set into the neural network for network training, calculating the loss function before the end of each round of network training to optimize the neural network, and obtaining the neural network training model after a total of 190-210 rounds of network training of network training. wherein, represents a saliency target image output in a multi-scale feature extraction module of a neural network, represents a label image, represents a weighted binary cross-entropy loss, represents a weighted intersection over union loss.

2. The method according to claim 1, wherein the method is a shared feature saliency object detection method based on lightweight network. The first MobileNetV2 backbone network performs five-layer serial feature extraction on the input color visible light image, each layer sequentially receives the output of the previous layer and generates a corresponding color feature map, and the first to fifth layers output a first color visible light feature map , a second color visible light feature map , a third color visible light feature map , a fourth color visible light feature map , and a fifth color visible light feature map . The second MobileNetV2 backbone network double-branch processing is five layers of serial feature extraction on the input pair of infrared images, each layer receiving the output of the previous layer in turn and generating a corresponding infrared image feature map, the first to fifth layers outputting a first infrared image feature map , a second infrared image feature map , a third infrared image feature map , a fourth infrared image feature map , and a fifth infrared image feature map , and five layers of serial feature extraction on the input pair of infrared images, each layer receiving the output of the previous layer in turn and generating a corresponding depth image feature map, the first to fifth layers outputting a first depth image feature map , a second depth image feature map , a third depth image feature map , a fourth depth image feature map , and a fifth depth image feature map .

3. The method of claim 1, wherein the method is based on a lightweight network of shared features. The common structure extraction module comprises five common structure extraction blocks with the same structure for refining common geometric structure information in infrared images and depth images, realizing cross-modal complementary fusion, and the common structure extraction block is composed of a first channel max-pooling layer, a second channel max-pooling layer, a first 1x1 convolution layer, a first Sigmoid activation layer, a first 3x3 convolution layer, a first BN layer, a second BN layer, a first ReLU activation layer, a second ReLU activation layer and a first 3x3 depth convolution layer.

4. The method according to claim 3, wherein the method is a shared feature saliency object detection method based on lightweight network. The common structure extraction block is implemented as follows: The first input of the common structure extraction block receives The second input receives The third input receives ;​ The input end of the first channel max pooling layer receives , and the output end outputs a feature map ; The input end of the second channel maximum pooling layer receives , and the output end outputs a feature map ; the element multiplication of and is performed to obtain a feature map ; the element multiplication of and is performed to obtain a feature map ; the splicing of and along the channel is performed to obtain a feature map ; the input end of the first 1x1 convolutional layer receives , the input end of the first Sigmoid activation layer receives the feature map output by the output end of the first 1x1 convolutional layer, and the output end of the first Sigmoid activation layer outputs a feature map ; the splicing of , and along the channel is performed to obtain a feature map ; the input end of the first 3x3 convolutional layer receives , the input end of the first BN layer receives the feature map output by the output end of the first 3x3 convolutional layer, the input end of the first ReLU activation layer receives the feature map output by the output end of the first BN layer, and the output end of the first ReLU activation layer outputs a feature map ; the splicing of and along the channel is performed to obtain a feature map ; the element addition of and is performed to obtain a feature map ; the input end of the first 3x3 depth convolutional layer receives , the input end of the second BN layer receives the feature map output by the output end of the first 3x3 depth convolutional layer, the input end of the second ReLU activation layer receives the feature map output by the output end of the second BN layer, and the output end of the second ReLU activation layer outputs a feature map ; wherein , , and and are in one-to-one correspondence; for the first common structure extraction block, the third receiving end has no input, at this time is completely the same as , wherein DT represents depth and infrared images of two modalities.

5. The method of claim 1, wherein the method is based on a lightweight network of shared features. The double-mode fusion module includes five structures same for the fusion strategy of semantic information, solves the double-mode fusion block of the high-quality fusion problem under three modes, and the double-mode fusion block is composed of a first global average pooling layer, a second global average pooling layer, a second 1*1 convolution layer, a third 1*1 convolution layer, a fourth 1*1 convolution layer, a fifth 1*1 convolution layer, a sixth 1*1 convolution layer, a seventh 1*1 convolution layer, a third ReLU activation layer, a fourth ReLU activation layer, a fifth ReLU activation layer, a second sigmoid activation layer, a third sigmoid activation layer, a fourth sigmoid activation layer, a fifth sigmoid activation layer, a third channel maximum pooling layer, a fourth channel maximum pooling layer, a first channel average pooling layer, a second channel average pooling layer, a second 3*3 deep convolution layer and a third BN layer.

6. The method according to claim 5, wherein the method is based on a lightweight network of shared features. The first input end of the bimodal fusion block receives The first input end of the bimodal fusion block receives The second receiving end receives The third input end receives The first input end receives ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​Element multiplication is performed to obtain a feature map ; the and are spliced along the channel to obtain a feature map ; the and are spliced along the channel to obtain a feature map ; the passes through a second 3*3 depth convolution layer, a third BN layer and a fifth ReLU activation layer in sequence to obtain a feature map ; wherein, , , and correspond to one by one; for the first dual-modal fusion block, the third receiving end has no input, at this time is completely the same as .

7. The method of claim 1, wherein the method is based on a lightweight network of shared features. The multi-scale feature refining module includes five structures same for the feature refining block of efficient transmission, fusion of semantic and detail information between different scales and levels, the feature refining block is composed of an eighth 1*1 convolution layer, a ninth 1*1 convolution layer, a tenth 1*1 convolution layer, an eleventh 1*1 convolution layer, a twelfth 1*1 convolution layer, a thirteenth 1*1 convolution layer, a third 3*3 deep convolution layer, a fourth 3*3 deep convolution layer, a fifth 3*3 deep convolution layer, a first hollow convolution layer with an expansion rate of 3, a second hollow convolution layer with an expansion rate of 5, a third hollow convolution layer with an expansion rate of 7, a second 3*3 convolution layer, a fourth BN layer, a sixth ReLU activation layer, a seventh ReLU activation layer, a first channel attention layer and a first up-sampling layer.

8. The method according to claim 7, wherein the method is a shared feature saliency object detection method based on lightweight network. No. The implementation process of the feature refining block is as follows: The first input of the feature refining block receives The second input terminal receives The third input terminal receives ;Will , , Element-wise addition yields the feature map. ;Will The inputs are fed into the eighth, ninth, tenth, eleventh, and twelfth 1×1 convolutional layers, respectively, to obtain the corresponding feature maps. Feature map Feature map Feature map Feature map ;Will and Perform element-wise addition to obtain the feature map. ;Will The feature map is obtained by sequentially passing through a third 3×3 depth convolutional layer and a first dilated convolutional layer. ;Will and Perform element-wise addition to obtain the feature map. ;Will The feature map is obtained by sequentially passing through a fourth 3×3 depth convolutional layer and a second dilated convolutional layer. ;Will and Perform element-wise addition to obtain the feature map. ;Will The feature map is obtained by sequentially passing through a fifth 3×3 depth convolutional layer and a third dilated convolutional layer. ;Will , , , By stitching along the channels, a feature map is obtained. ;Will The feature map is obtained by sequentially passing through the second 3×3 convolutional layer, the fourth BN layer, the sixth ReLU activation layer, and the first channel attention layer. ;Will and Element-wise addition yields the feature map. ; The input of the seventh ReLU activation layer receives The feature map output from the seventh ReLU activation layer is also the feature map of the seventh ReLU activation layer. Feature map of the first output of the multi-scale feature extraction block. ;Will The feature map obtained after the thirteenth 1×1 convolutional layer and the first upsampling layer is also the... Feature map at the second output of the multi-scale feature extraction block ;in, , ,and and One-to-one correspondence; for the fifth feature refinement block, the third receiver has no input, at this time only for , When adding elements, where, As the final salient target image, RDT represents three modes: color visible light, depth, and infrared.

9. The method of claim 1, wherein the method is based on a lightweight network of shared features. The process of predicting the salient target image of the test image pair by using the neural network trained model is: firstly, a pair of original color visible light images, original infrared images and original depth images corresponding to the original color visible light images are selected at random; then, the pair of original color visible light images, original infrared images and original depth images are down-sampled, and the size of the down-sampled images is , and the down-sampled images are taken as the test image pair; finally, the test image pair is input into the trained neural network model, and the corresponding salient target image is predicted.

Images