A salient object detection method based on lightweight neural network

Abstract

The present invention discloses a significant target detection method based on a lightweight neural network, which builds a lightweight neural network, which is mainly composed of a high-level lightweight fusion module, a low-level lightweight fusion module, and a two-stage decoder; the lightweight neural network is composed of a feature extraction module, a high-level lightweight fusion module, a low-level lightweight fusion module, and a two-stage decoder: the feature extraction module uses the MobileViT-XS backbone network to extract features from color visible light images and infrared images; the high-level lightweight fusion module and the low-level lightweight fusion module use convolution to extract local information, and use an improved cross-attention operation to narrow the gap between visible light and infrared modalities; the two-stage decoder densely connects three high-level features to generate a guiding feature, which guides the fusion of two low-level features to produce a final result. The present invention can effectively improve the accuracy of significant target detection with low parameter amount and calculation amount.

Description

Technical Field

[0001] The present invention relates to the technical field of salient target detection, and in particular to a salient target detection method based on a lightweight neural network. Background Art

[0002] Salient object detection aims to capture and segment salient objects in images or videos. As an important preprocessing step, salient object detection has been widely used in computer vision and image processing tasks such as image segmentation, object tracking, image retrieval, and image quality assessment. In recent years, convolutional / deep neural networks have pushed the performance of salient object detection to a new level due to their powerful learning ability and excellent performance in feature extraction. However, in some extreme scenarios (such as low illumination and cluttered scenes), it is often difficult to extract valuable information from color visible light images alone, which affects the effectiveness of salient object detection models under complex conditions. Unlike color visible light images, thermal infrared cameras can capture thermal information that is less affected by environmental factors such as darkness and bad weather, thereby reflecting temperature differences, boundaries, and geometric shapes. Therefore, deploying thermal infrared equipment to collect thermal information and using color visible light and infrared image salient object detection models to record objects can improve the model's perception ability in more scenarios.

[0003] Salient object detection in color visible light and depth images with auxiliary depth images and color visible light and infrared images with infrared images have been developed, using a wide range of depth sensors and infrared cameras as additional modality information. Depth information contains rich spatial structure and 3D layout information, but it is unreliable to provide useful information for salient object detection in some extreme environments (e.g., poor lighting and cluttered scenes). Since infrared images can reflect the thermal radiation of the surface of objects, they are naturally complementary to color visible light images in these extreme environments. In recent years, research on salient object detection in color visible light and infrared images has received increasing attention.

[0004] On the other hand, existing RGB-T salient object detection models have made significant breakthroughs in accuracy, but these high-precision models usually require high computational costs and large model sizes to process multimodal information, which means they can only be deployed in facilities with high-performance graphics processing units. In terms of computing latency and bandwidth costs, they cannot meet the needs of some practical scenarios, such as autonomous driving, monitoring the Internet of Things, etc.

[0005] At present, there are some lightweight RGB-T salient object detection models, which are mainly based on convolutional neural networks (CNNs). These methods usually use a lightweight backbone network to extract features from two modalities, and then perform multi-modal and multi-level feature fusion through various carefully designed modules. Although these methods have successfully compressed the model size and can be deployed on some resource-constrained devices, it is still difficult to strike a balance between performance and complexity. Therefore, how to enhance the prediction performance of lightweight RGB-T salient object detection models while maintaining a low model size has become an urgent problem to be solved. Compared with the CNN structure, the Transformer can model long-range dependencies, which helps to locate salient objects and suppress noise. However, compared with CNN, the Transformer architecture has a larger computational cost and longer response time. Therefore, in the lightweight RGB-T SOD task, it is particularly critical to combine the advantages of Transformer and CNN in the feature extraction and fusion stages while maintaining a small model size. Summary of the invention

[0006] The present invention provides a salient target detection method based on a lightweight neural network, which can effectively improve the accuracy of salient target detection with a low amount of parameters and calculations.

[0007] The present invention provides a salient target detection method based on a lightweight neural network. First, a training set including several pairs of color visible light images and corresponding infrared images is constructed, and a lightweight neural network is built; secondly, the several pairs of color visible light images and corresponding infrared images in the training set are input into the neural network for multiple rounds of network training, and a neural network training model is obtained after the network training is completed; the neural network training model is used again to predict the test image pair, and the salient target image of the test image pair is predicted. The lightweight neural network is composed of a feature extraction module, a high-level lightweight fusion module, a low-level lightweight fusion module, and a two-stage decoder: The feature extraction module uses the MobileViT-XS backbone network to extract features from color visible light images and infrared images; The high-level lightweight fusion module and the low-level lightweight fusion module use convolution to extract local information and use an improved cross-attention operation to narrow the gap between visible and infrared modalities; The two-stage decoder densely connects the three high-level features to generate a guiding feature, which guides the fusion of the two low-level features to produce the final result.

[0008] Furthermore, it also includes: The feature extraction module includes two MobileViT-XS backbone networks; the first layer of the first MobileViT-XS backbone network receives a color visible light image, and the output feature map is recorded as , the second layer receiving of the first MobileViT-XS backbone network , the output feature map is recorded as , the third layer receiving of the first MobileViT-XS backbone network , the output feature map is recorded as , the fourth layer receiving of the first MobileViT-XS backbone network , the output feature map is recorded as , the fifth layer receiving of the first MobileViT-XS backbone network , the output feature map is recorded as ; The first layer of the second MobileViT-XS backbone network receives the infrared image, and the output feature map is recorded as , the second layer receiving of the second MobileViT-XS backbone network , the output feature map is recorded as , the third layer receiving of the second MobileViT-XS backbone network , the output feature map is recorded as , the fourth layer receiving of the second MobileViT-XS backbone network , the output feature map is recorded as , the fifth layer receiving of the second MobileViT-XS backbone network , the output feature map is recorded as ; The low-level lightweight fusion module includes two low-level lightweight fusion blocks with the same structure; the first low-level lightweight fusion block receives and , the output feature map is recorded as ; The second low-level lightweight fusion block receives and , the output feature map is recorded as ; The high-level lightweight fusion module includes three high-level lightweight fusion blocks with the same structure; the first high-level lightweight fusion block receives and , the output feature map is recorded as ; The second high-level lightweight fusion block receives and , the output feature map is recorded as ; The third high-level lightweight fusion block receives and , the output feature map is recorded as ; The two-stage decoder receives , , , and , the output feature map is recorded as , , , , and As the final salient target image.

[0009] Furthermore, it also includes: The construction process of the training set is as follows: select at least 1000 pairs of original color visible light images and their corresponding original infrared images; then downsample each original color visible light image and its corresponding original infrared image, and the size of the downsampled image is ; Then all the sizes are The color visible light images and the corresponding infrared images constitute the training set; among them, ; The process of obtaining the neural network training model is as follows: each pair of color visible light images and their corresponding infrared images in the training set are input into the neural network for network training, and the loss function is calculated before each round of network training ends. To optimize the neural network, a neural network training model was obtained after a total of 150 rounds of network training; among them, , represents the final salient target image output by the neural network, represents the label image, Represents the first A coarse salient target image, represents the binary cross entropy loss, represents the intersection-and-combination loss.

[0010] Furthermore, it also includes: The process of using the neural network training model to predict the test image pair to obtain the salient target image is as follows: arbitrarily select a pair of original color visible light images and the corresponding original infrared images; then downsample the pair of original color visible light images and original infrared images, and the size of the downsampled image is , and used as a test image pair; the test image pair is then input into the trained neural network model to predict the corresponding salient target image.

[0011] Furthermore, it also includes: The structures of the three high-level lightweight fusion blocks are the same. High-level lightweight fusion blocks ( ), The input of the convolutional layer receives ( ), the input of the Batch Normalization layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and the output feature map is recorded as ; another The input of the convolutional layer receives The input of the Batch Normalization layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and the output feature map is recorded as .

[0012] Furthermore, it also includes: The structures of the two low-level lightweight fusion blocks are the same. Low-level lightweight fusion blocks , The input of the convolutional layer receives The input of the Batch Normalization layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and the output feature map is recorded as ; another The input of the convolutional layer receives The input of the Batch Normalization layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and the output feature map is recorded as .

[0013] Furthermore, it also includes: Two-stage decoder reception , , , and ,right Perform upsampling operation, the output feature map is Perform channel connection operation and input the obtained feature map into The input of the convolutional layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the Batch Normalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as .

[0014] Furthermore, it also includes: right and To expand the Format converted to format, where ,and , and the obtained feature map is recorded as and ; one The input of the fully connected layer receives , the input of the ReLU function receives the The feature map output by the fully connected layer is recorded as ; one The input of the fully connected layer receives , the input of the ReLU function receives the The feature map output by the fully connected layer is recorded as ; right and Perform channel connection operation and record the obtained feature map as ; one The input of the fully connected layer receives , the output feature map is recorded as ; one The input of the fully connected layer receives , the input of the Softmax function receives the The feature map output by the fully connected layer is recorded as ; right and Perform element-wise multiplication and The N dimensions in the format are summed and the resulting feature map is recorded as ; right and Perform element-wise multiplication, The input of the fully connected layer receives the result of the element-wise multiplication operation, and the output feature map is recorded as ,right and Perform element addition operation and record the obtained feature map as ;one The input of the fully connected layer receives The input of the SiLU function receives the The feature map output at the output of the fully connected layer is a The input end of the fully connected layer receives the feature map output by the output end of the SiLU function, and the output feature map is recorded as ,right and Perform element addition operation and record the obtained feature map as ; right and Perform element-wise multiplication, The input of the fully connected layer receives the result of the element-wise multiplication operation, and the output feature map is recorded as ,right and Perform element addition operation and record the obtained feature map as ,one The input of the fully connected layer receives The input of the SiLU function receives the The feature map output at the output of the fully connected layer is a The input end of the fully connected layer receives the feature map output by the output end of the SiLU function, and the output feature map is recorded as ,right and Perform element addition operation and record the obtained feature map as ; Repeat from and generate and The process of and This will be used as the next step in the and The final output is recorded as and ; right and Folding operation is performed by Format converted to Format, the obtained feature map is recorded as and ,in, , and with and The same size; right and Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as ,right and Perform element addition operation, and the obtained feature map is recorded as ; right and Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as ,right and Perform element addition operation, and the obtained feature map is recorded as ; right and Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as , That is the The feature map output by the output of the high-level lightweight fusion block.

[0015] Furthermore, it also includes: right and To expand the Format converted to format, where ,and , and the obtained feature map is recorded as and ; one The input of the fully connected layer receives , the input of the ReLU function receives the The feature map output by the fully connected layer is recorded as ; right and Perform channel connection operation and record the obtained feature map as ; one The input of the fully connected layer receives , the output feature map is recorded as ; one The input of the fully connected layer receives , the input of the Softmax function receives the The feature map output by the fully connected layer is recorded as ; right and Perform element-wise multiplication and The N dimensions in the format are summed and the resulting feature map is recorded as ; right and Perform element-wise multiplication, The input of the fully connected layer receives the result of the element-wise multiplication operation, and the output feature map is recorded as ,right and Perform element addition operation and record the obtained feature map as ,one The input of the fully connected layer receives The input of the SiLU function receives the The feature map output at the output of the fully connected layer is a The input end of the fully connected layer receives the feature map output by the output end of the SiLU function, and the feature map output by the output end is recorded as ,right and Perform element addition operation and record the obtained feature map as ; Repeat from and generate The process of This will be used as the next step in the The final output is recorded as ; right Folding operation is performed by Format converted to Format, the obtained feature map is recorded as ,in , and with The same size; right , and Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as , That is the The feature map output by the output of the low-level lightweight fusion block.

[0016] Furthermore, it also includes: right and Perform upsampling operation, the output feature map is Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the BatchNormalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as ; Will Input to At the input end of the convolutional layer, the output features are upsampled, and the resulting feature map is recorded as , The first feature map output as the output of the two-stage decoder; right Perform upsampling operation, the output feature map is Perform channel connection operation, and the obtained feature map is recorded as ,Will Input to The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the BatchNormalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as ; Will Input to At the input end of the convolutional layer, the output features are upsampled, and the resulting feature map is recorded as , The second feature map output as the output of the two-stage decoder; right Perform upsampling operation, the output feature map is Perform channel connection operation, and the obtained feature map is recorded as ,Will Input to The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the BatchNormalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as ; Will Input to At the input end of the convolutional layer, the output features are upsampled, and the resulting feature map is recorded as , The third feature map output as the output of the two-stage decoder; right Perform upsampling operation, the output feature map is Perform channel connection operation, and the obtained feature map is recorded as ,Will Input to The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the BatchNormalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, performs upsampling on the output features, and the obtained feature map is recorded as , The fourth feature map is output as the output of the two-stage decoder.

[0017] The beneficial effects of the present invention are: 1) The lightweight neural network constructed by the method of the present invention adopts the MobileViT-XS backbone network for feature extraction, and extracts features from color visible light images and infrared images; then a high-level lightweight fusion module and a low-level lightweight fusion module are adopted, convolution is used to summarize local information, and an improved cross-attention operation is used to narrow the gap between visible light and infrared modalities; finally, a two-stage decoder is used to densely connect the three high-level features to generate a guiding feature, which guides the fusion of two low-level features to produce the final result.

[0018] 2) The method of the present invention combines the respective advantages of CNN and Transformer structures, designs low-level lightweight fusion modules and high-level lightweight fusion modules of different complexities, and fully captures global and local information with fewer parameters.

[0019] 3) To address the problem that multimodal features are difficult to fuse with high quality, the method of the present invention adopts a two-stage decoder in the constructed neural network. The decoder can mine high-level feature information and make full use of its semantic information to obtain better results, thereby effectively improving the accuracy of salient target detection in color visible light and infrared images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 It is the overall implementation framework diagram of the method of the present invention; Figure 2 A schematic diagram of the composition structure of a neural network constructed by the method of the present invention; Figure 3 A schematic diagram of the composition structure of a high-level lightweight fusion block in a neural network constructed by the method of the present invention; Figure 4 A schematic diagram of the structure of the remaining parts of the high-level lightweight fusion block in the neural network constructed by the method of the present invention; Figure 5 A schematic diagram of the composition structure of a low-level lightweight fusion block in a neural network constructed by the method of the present invention; Figure 6 A schematic diagram of the structure of a two-stage decoder in a neural network constructed by the method of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0022] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0023] The present invention proposes a salient target detection method based on a lightweight neural network, and its overall implementation framework is shown in the following figure: Figure 1As shown in FIG, the method first constructs a training set containing several pairs of color visible light images and their corresponding infrared images, and builds a lightweight neural network; secondly, the several pairs of color visible light images and their corresponding infrared images in the training set are input into the neural network for multiple rounds of network training, and a neural network training model is obtained after the network training is completed; the neural network training model is used again to predict the test image pairs, and the salient target images of the test image pairs are predicted, such as Figure 2 The lightweight neural network shown is mainly composed of a feature extraction module, a high-level lightweight fusion module, a low-level lightweight fusion module, and a two-stage decoder. The feature extraction module uses the MobileViT-XS backbone network to extract features from color visible light images and infrared images; High-level lightweight fusion modules and low-level lightweight fusion modules use convolution to extract local information and use improved cross-attention operations to narrow the gap between visible and infrared modalities; The two-stage decoder densely connects the three high-level features to generate a guiding feature, which guides the fusion of the two low-level features to produce the final result.

[0024] The feature extraction module includes two MobileViT-XS backbone networks; the first layer of the first MobileViT-XS backbone network receives the color visible light image, and the output feature map is recorded as , the second layer receiving of the first MobileViT-XS backbone network , the output feature map is recorded as , the third layer receiving of the first MobileViT-XS backbone network , the output feature map is recorded as , the fourth layer receiving of the first MobileViT-XS backbone network , the output feature map is recorded as , the fifth layer receiving of the first MobileViT-XS backbone network , the output feature map is recorded as ; The first layer of the second MobileViT-XS backbone network receives the infrared image, and the output feature map is recorded as , the second layer receiving of the second MobileViT-XS backbone network , the output feature map is recorded as , the third layer receiving of the second MobileViT-XS backbone network , the output feature map is recorded as , the fourth layer receiving of the second MobileViT-XS backbone network , the output feature map is recorded as , the fifth layer receiving of the second MobileViT-XS backbone network , the output feature map is recorded as ; Among them, the MobileViT-XS backbone network is an existing structural framework, and its network structure has been made public, as recorded in the document MehtaS, Rastegari M. Mobilevit: light-weight, general-purpose, and mobile-friendlyvision transformer[J]. arXiv preprint arXiv:2110.02178, 2021.

[0025] The low-level lightweight fusion module includes two low-level lightweight fusion blocks with the same structure; the receiving and , the output feature map is recorded as ; The second low-level lightweight fusion block receives and , the output feature map is recorded as .

[0026] The high-level lightweight fusion module includes three high-level lightweight fusion blocks with the same structure; the first high-level lightweight fusion block receives and , the output feature map is recorded as ; The second high-level lightweight fusion block receives and , the output feature map is recorded as ; The third high-level lightweight fusion block receives and , the output feature map is recorded as .

[0027] Two-stage decoder reception , , , and , the output feature map is recorded as , , , , and As the final salient target image.

[0028] In a specific embodiment, the training set is constructed by selecting at least 1000 pairs of original color visible light images and their corresponding original infrared images; then downsampling each original color visible light image and its corresponding original infrared image, and the size of the downsampled image is ; Then all the sizes are The color visible light images and the corresponding infrared images constitute the training set; among them, .

[0029] In a specific embodiment, the process of obtaining the neural network training model is as follows: each pair of color visible light images and the corresponding infrared images in the training set are input into the neural network for network training, and the loss function is calculated before each round of network training ends. To optimize the neural network, a neural network training model was obtained after a total of 150 rounds of network training; , represents the final salient target image output by the neural network, represents the label image, Represents the first A coarse salient target image, represents the binary cross entropy (BCE) loss, represents the Intersection over Union (IoU) loss.

[0030] In a specific embodiment, a neural network training model is used to predict a test image pair, and the process of predicting a salient target image of the test image pair is as follows: arbitrarily select a pair of original color visible light images and the corresponding original infrared images; then downsample the pair of original color visible light images and the original infrared images, and the size of the downsampled images is , and used as a test image pair; then the test image pair is input into the trained neural network model to predict the corresponding salient target image; where, .

[0031] The structures of the three high-level lightweight fusion blocks are the same, except for the input and output. In a specific embodiment, Figure 3 and Figure 4 As shown, for the High-level lightweight fusion blocks ( ), The input of the convolutional layer receives ( ), the input of the Batch Normalization layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and the output feature map is recorded as .

[0032] another The input of the convolutional layer receives The input of the Batch Normalization layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and the output feature map is recorded as .

[0033] right and To expand the Format converted to Format, where ,and , and the obtained feature map is recorded as and .

[0034] one The input of the fully connected layer receives , the input of the ReLU function receives the The feature map output by the fully connected layer is recorded as .

[0035] one The input of the fully connected layer receives , the input of the ReLU function receives the The feature map output by the fully connected layer is recorded as .

[0036] right and Perform channel connection operation (Concatenation), and record the obtained feature map as .

[0037] one The input of the fully connected layer receives , the output feature map is recorded as .

[0038] one The input of the fully connected layer receives , the input of the Softmax function receives the The feature map output by the fully connected layer is recorded as .

[0039] right and Perform element-wise multiplication and The N dimensions in the format are summed and the resulting feature map is recorded as .

[0040] right and Perform element-wise multiplication, The input of the fully connected layer receives the result of the element-wise multiplication operation, and the output feature map is recorded as .right and Perform element addition operation and record the obtained feature map as .one The input of the fully connected layer receives The input of the SiLU function receives the The feature map output at the output of the fully connected layer is a The input end of the fully connected layer receives the feature map output by the output end of the SiLU function, and the output feature map is recorded as .right and Perform element addition operation and record the obtained feature map as .

[0041] right and Perform element-wise multiplication, The input of the fully connected layer receives the result of the element-wise multiplication operation, and the output feature map is recorded as .right and Perform element addition operation and record the obtained feature map as .one The input of the fully connected layer receives The input of the SiLU function receives the The feature map output at the output of the fully connected layer is a The input end of the fully connected layer receives the feature map output by the output end of the SiLU function, and the output feature map is recorded as .right and Perform element addition operation and record the obtained feature map as .

[0042] like Figure 3 As shown, from and generate and The process will be repeated once more, with the last generated and This will be used as the next step in the and The final output is recorded as and .

[0043] like Figure 4 As shown, and Folding operation is performed by Format converted to Format, the obtained feature map is recorded as and ,in , and with and The same size.

[0044] right and Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as .right and Perform element addition operation, and the obtained feature map is recorded as .

[0045] right and Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as .right and Perform element addition operation, and the obtained feature map is recorded as .

[0046] right and Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as . That is the The feature map output by the output of the high-level lightweight fusion block.

[0047] Here, element multiplication, element addition, element subtraction, and channel concatenation are all common operations in neural networks. Figure 3 , Figure 4 The BN layer is an abbreviation of the Batch Normalization layer.

[0048] The structures of the two low-level lightweight fusion blocks are the same, except for the input and output. In a specific embodiment, Figure 5 As shown, for the Low-level lightweight fusion blocks , The input of the convolutional layer receives The input of the BatchNormalization layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and the output feature map is recorded as .

[0049] another The input of the convolutional layer receives The input of the Batch Normalization layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and the output feature map is recorded as .

[0050] right and To expand the Format converted to Format, where ,and , and the obtained feature map is recorded as and .

[0051] one The input of the fully connected layer receives , the input of the ReLU function receives the The feature map output by the fully connected layer is recorded as .

[0052] right and Perform channel connection operation (Concatenation), and record the obtained feature map as .

[0053] one The input of the fully connected layer receives , the output feature map is recorded as .

[0054] one The input of the fully connected layer receives , the input of the Softmax function receives the The feature map output by the fully connected layer is recorded as .

[0055] right and Perform element-wise multiplication and The N dimensions in the format are summed and the resulting feature map is recorded as .

[0056] right and Perform element-wise multiplication, The input of the fully connected layer receives the result of the element-wise multiplication operation, and the output feature map is recorded as .right and Perform element addition operation and record the obtained feature map as .one The input of the fully connected layer receives The input of the SiLU function receives the The feature map output at the output of the fully connected layer is a The input end of the fully connected layer receives the feature map output by the output end of the SiLU function, and the output feature map is recorded as .right and Perform element addition operation and record the obtained feature map as .

[0057] like Figure 3 As shown, from and generate The process will be repeated once more, with the last generated This will be used as the next step in the The final output is recorded as .

[0058] like Figure 4 As shown, Folding operation is performed by Format converted to Format, the obtained feature map is recorded as ,in , and with The same size.

[0059] right , and Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as . That is the The feature map output by the output of the low-level lightweight fusion block.

[0060] Here, element multiplication, element addition, element subtraction, and channel concatenation are all common operations in neural networks. Figure 5 The BN layer is an abbreviation of the Batch Normalization layer.

[0061] like Figure 6 As shown, the two-stage decoder receives , , , and ,right Perform upsampling operation, the output feature map is Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the Batch Normalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as .

[0062] right and Perform upsampling operation, the output feature map is Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the BatchNormalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as .

[0063] Will Input to At the input end of the convolutional layer, the output features are upsampled, and the resulting feature map is recorded as , The first feature map outputted as the output of the two-stage decoder.

[0064] right Perform upsampling operation, the output feature map is Perform channel connection operation, and the obtained feature map is recorded as .Will Input to The input end of the convolution layer, the input end of the Batch Normalization layer receives the feature map output by the output end of the 1×1 convolution layer, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the BatchNormalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as .

[0065] Will Input to At the input end of the convolutional layer, the output features are upsampled, and the resulting feature map is recorded as , The second feature map outputted as the output of the two-stage decoder.

[0066] right Perform upsampling operation, the output feature map is Perform channel connection operation, and the obtained feature map is recorded as .Will Input to The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the BatchNormalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as .

[0067] Will Input to At the input end of the convolutional layer, the output features are upsampled, and the resulting feature map is recorded as , The third feature map output as the output of the two-stage decoder.

[0068] right Perform upsampling operation, the output feature map is Perform channel connection operation, and the obtained feature map is recorded as .Will Input to The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the BatchNormalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, performs upsampling on the output features, and the obtained feature map is recorded as , The fourth feature map is output as the output of the two-stage decoder.

[0069] In order to further illustrate the feasibility and effectiveness of the method of the present invention, the method of the present invention was tested.

[0070] In this embodiment, the method of the present invention is used to test the VT5000 data set. The training set in VT5000 includes 2500 pairs of color visible light images and infrared images, and the test set in VT5000 has a total of 2500 pairs of color visible light images and infrared images, that is, 2500 pairs of test image pairs.

[0071] In this embodiment, four commonly used objective parameters are selected to evaluate the performance of the method of the present invention, namely S-measure, E-measure, F-measure, and Mean Absolute Error (MAE). Table 1 shows the correlation between the salient target image and the label image obtained by the method of the present invention on the VT5000 dataset.

[0072] Table 1 S-measure, E-measure, F-measure, and Mean Absolute Error (MAE) between the salient target image and the label image obtained by the method of the present invention on the VT5000 dataset From the results given in Table 1, it can be found that the method of the present invention has achieved higher S-measure, E-measure, F-measure and lower MAE on the existing color visible light and infrared image salient target detection datasets, and has lower parameter and computational complexity. This shows that the salient target image obtained by the method of the present invention is closer to the label image, and the method of the present invention can effectively complete the salient target detection of color visible light and infrared images with low parameter and computational complexity.

[0073] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A salient target detection method based on a lightweight neural network, firstly constructing a training set including several pairs of color visible light images and corresponding infrared images, and building a lightweight neural network; secondly, inputting several pairs of color visible light images and corresponding infrared images in the training set into the neural network for several network trainings, and obtaining a neural network training model after the network training is completed; and again using the neural network training model to predict the test image pairs to obtain a salient target image, characterized in that: The lightweight neural network is composed of a feature extraction module, a high-level lightweight fusion module, a low-level lightweight fusion module, and a two-stage decoder: The feature extraction module uses the MobileViT-XS backbone network to extract features from color visible light images and infrared images; High-level lightweight fusion modules and low-level lightweight fusion modules use convolution to extract local information and use improved cross-attention operations to narrow the gap between visible and infrared modalities; The two-stage decoder densely connects the three high-level features to generate a guiding feature, which guides the fusion of the two low-level features to produce the final result.

2. The method for detecting salient objects based on a lightweight neural network according to claim 1, characterized in that: The feature extraction module includes two MobileViT-XS backbone networks; the first layer of the first MobileViT-XS backbone network receives a color visible light image, and the output feature map is recorded as , the second layer receiving of the first MobileViT-XS backbone network , the output feature map is recorded as , the third layer receiving of the first MobileViT-XS backbone network , the output feature map is recorded as , the fourth layer receiving of the first MobileViT-XS backbone network , the output feature map is recorded as , the fifth layer receiving of the first MobileViT-XS backbone network , the output feature map is recorded as ; The first layer of the second MobileViT-XS backbone network receives the infrared image, and the output feature map is recorded as , the second layer receiving of the second MobileViT-XS backbone network , the output feature map is recorded as , the third layer receiving of the second MobileViT-XS backbone network , the output feature map is recorded as , the fourth layer receiving of the second MobileViT-XS backbone network , the output feature map is recorded as , the fifth layer receiving of the second MobileViT-XS backbone network , the output feature map is recorded as ; The low-level lightweight fusion module includes two low-level lightweight fusion blocks with the same structure; the first low-level lightweight fusion block receives and , the output feature map is recorded as ; The second low-level lightweight fusion block receives and , the output feature map is recorded as ; The high-level lightweight fusion module includes three high-level lightweight fusion blocks with the same structure; the first high-level lightweight fusion block receives and , the output feature map is recorded as ; The second high-level lightweight fusion block receives and , the output feature map is recorded as ; The third high-level lightweight fusion block receives and , the output feature map is recorded as ; The two-stage decoder receives , , , and , the output feature map is recorded as , , , , and As the final salient target image.

3. The method for detecting salient objects based on a lightweight neural network according to claim 2, characterized in that: The construction process of the training set is as follows: select at least 1000 pairs of original color visible light images and their corresponding original infrared images; then downsample each original color visible light image and its corresponding original infrared image, and the size of the downsampled image is ; Then all the sizes are The color visible light images and the corresponding infrared images constitute the training set; among them, ; The process of obtaining the neural network training model is as follows: each pair of color visible light images and their corresponding infrared images in the training set are input into the neural network for network training, and the loss function is calculated before each round of network training ends. To optimize the neural network, a neural network training model was obtained after a total of 150 rounds of network training; among them, , represents the final salient target image output by the neural network, represents the label image, Represents the first A coarse salient target image, represents the binary cross entropy loss, represents the intersection-and-combination loss.

4. The method for detecting salient objects based on a lightweight neural network according to claim 2, characterized in that: The process of using the neural network training model to predict the test image pair to obtain the salient target image is as follows: arbitrarily select a pair of original color visible light images and the corresponding original infrared images; then downsample the pair of original color visible light images and original infrared images, and the size of the downsampled image is , and used as a test image pair; the test image pair is then input into the trained neural network model to predict the corresponding salient target image.

5. The method for detecting salient objects based on a lightweight neural network according to claim 2, characterized in that: The structures of the three high-level lightweight fusion blocks are the same. High-level lightweight fusion blocks ( ), The input of the convolutional layer receives ( ), the input of the Batch Normalization layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and the output feature map is recorded as ; another The input of the convolutional layer receives The input of the Batch Normalization layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and the output feature map is recorded as .

6. The method for detecting salient objects based on a lightweight neural network according to claim 2, characterized in that: The structures of the two low-level lightweight fusion blocks are the same. Low-level lightweight fusion blocks , The input of the convolutional layer receives The input of the Batch Normalization layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and the output feature map is recorded as ; another The input of the convolutional layer receives The input of the Batch Normalization layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and the output feature map is recorded as .

7. The method for detecting salient objects based on a lightweight neural network according to claim 2, characterized in that: Two-stage decoder reception , , , and ,right Perform upsampling operation, the output feature map is Perform channel connection operation and input the obtained feature map into The input of the convolutional layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the Batch Normalization layer receives the feature map output by the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as .

8. The method for detecting salient objects based on a lightweight neural network according to claim 7, characterized in that: Also includes: right and To expand the Format converted to Format, where ,and , and the obtained feature map is recorded as and ; one The input of the fully connected layer receives , the input of the ReLU function receives the The feature map output by the fully connected layer is recorded as ; one The input of the fully connected layer receives , the input of the ReLU function receives the The feature map output by the fully connected layer is recorded as ; right and Perform channel connection operation and record the obtained feature map as ; one The input of the fully connected layer receives , the output feature map is recorded as ; one The input of the fully connected layer receives , the input of the Softmax function receives the The feature map output by the fully connected layer is recorded as ; right and Perform element-wise multiplication and The N dimensions in the format are summed and the resulting feature map is recorded as ; right and Perform element-wise multiplication, The input of the fully connected layer receives the result of the element-wise multiplication operation, and the output feature map is recorded as ,right and Perform element addition operation and record the obtained feature map as ;one The input of the fully connected layer receives The input of the SiLU function receives the The feature map output at the output of the fully connected layer is a The input end of the fully connected layer receives the feature map output by the output end of the SiLU function, and the output feature map is recorded as ,right and Perform element addition operation and record the obtained feature map as ; right and Perform element-wise multiplication, The input of the fully connected layer receives the result of the element-wise multiplication operation, and the output feature map is recorded as ,right and Perform element addition operation and record the obtained feature map as ,one The input of the fully connected layer receives The input of the SiLU function receives the The feature map output at the output of the fully connected layer is a The input end of the fully connected layer receives the feature map output by the output end of the SiLU function, and the output feature map is recorded as ,right and Perform element addition operation and record the obtained feature map as ; Repeat from and generate and The process of which the last generated and This will be used as the next step in the and The final output is recorded as and ; right and Folding operation is performed by Format converted to Format, the obtained feature map is recorded as and ,in, , and with and The same size; right and Perform channel connection operation and input the obtained feature map into The input of the convolutional layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as ,right and Perform element addition operation, and the obtained feature map is recorded as ; right and Perform channel connection operation and input the obtained feature map into The input of the convolutional layer receives the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as ,right and Perform element addition operation, and the obtained feature map is recorded as ; right and Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as , That is the The feature map output by the output of the high-level lightweight fusion block.

9. The method for detecting salient objects based on a lightweight neural network according to claim 8, characterized in that: Also includes: right and To expand the Format converted to format, where ,and , and the obtained feature map is recorded as and ; one The input of the fully connected layer receives , the input of the ReLU function receives the The feature map output by the fully connected layer is recorded as ; right and Perform channel connection operation and record the obtained feature map as ; one The input of the fully connected layer receives , the output feature map is recorded as ; one The input of the fully connected layer receives , the input of the Softmax function receives the The feature map output by the fully connected layer is recorded as ; right and Perform element-wise multiplication and The N dimensions in the format are summed and the resulting feature map is recorded as ; right and Perform element-wise multiplication, The input of the fully connected layer receives the result of the element-wise multiplication operation, and the output feature map is recorded as ,right and Perform element addition operation and record the obtained feature map as ,one The input of the fully connected layer receives The input of the SiLU function receives the The feature map output at the output of the fully connected layer is a The input end of the fully connected layer receives the feature map output by the output end of the SiLU function, and the feature map output by the output end is recorded as ,right and Perform element addition operation and record the obtained feature map as ; Repeat from and generate The process of which the last generated This will be used as the next step in the The final output is recorded as ; right Folding operation is performed by Format converted to Format, the obtained feature map is recorded as ,in , and with The same size; right , and Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as , That is the The feature map output by the output of the low-level lightweight fusion block.

10. The method for detecting salient objects based on a lightweight neural network according to claim 9, characterized in that: Also includes: right and Perform upsampling operation, the output feature map is Perform channel connection operation and input the obtained feature map into The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the Batch Normalization layer receives the feature map output by the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the BatchNormalization layer. The output feature map is recorded as ; Will Input to At the input end of the convolutional layer, the output features are upsampled, and the resulting feature map is recorded as , The first feature map output as the output of the two-stage decoder; right Perform upsampling operation, the output feature map is Perform channel connection operation, and the obtained feature map is recorded as ,Will Input to The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the BatchNormalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as ; Will Input to At the input end of the convolutional layer, the output features are upsampled, and the resulting feature map is recorded as , The second feature map output as the output of the two-stage decoder; right Perform upsampling operation, the output feature map is Perform channel connection operation, and the obtained feature map is recorded as ,Will Input to The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the BatchNormalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The output feature map is recorded as ; Will Input to At the input end of the convolutional layer, the output features are upsampled, and the resulting feature map is recorded as , The third feature map output as the output of the two-stage decoder; right Perform upsampling operation, the output feature map is Perform channel connection operation, and the obtained feature map is recorded as ,Will Input to The input of the convolutional layer and the input of the Batch Normalization layer receive the The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolution layer receives the feature map output by the output end of the ReLU function, and the input end of the BatchNormalization layer receives the feature map output by the output end of the ReLU function. The output end of the convolutional layer outputs the feature map, and the input end of the ReLU function receives the feature map output by the output end of the Batch Normalization layer. The input end of the convolutional layer receives the feature map output by the output end of the ReLU function, and performs upsampling operation on the output features. The obtained feature map is recorded as , The fourth feature map is output as the output of the two-stage decoder.

Images