Visible light-thermal infrared salient target detection method based on conditional generative adversarial network

Abstract

The present invention discloses a visible light-thermal infrared salient target detection method based on conditional generative adversarial network. The present invention designs two generators based on the conditional generative adversarial network model, and respectively improves the recall rate and the precision rate based on adversarial learning and makes them reach a balance, so as to obtain a more refined salient target prediction result; integrates high-level semantic features through a global context information module, and the integrated features contain richer global context information as guiding features of each layer of the decoder; the collaborative attention model of the present invention fuses the visible light modality and thermal infrared modality features, while further highlighting the foreground target area and suppressing the influence of redundant background noise. The salient map of the present invention has a higher confidence than other advanced methods.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision processing technology, and in particular relates to an RGB-T salient object detection method based on a conditional generative adversarial network. Background Art

[0002] Salient object detection is an important topic in the field of computer vision. This task has been widely used in various fields of computer vision, such as object segmentation, image retrieval, video object tracking, etc. At present, salient object detection in visible light images has been improved to a very high level in experimental environments. However, in real scenes, visible light imaging is often affected by factors such as bad weather, complex background, and poor lighting. Therefore, it is difficult to accurately detect salient objects using only visible light images, which greatly limits its application.

[0003] In recent years, with the diversification of imaging and its information description methods, attempts to collaboratively utilize multimodal image information such as depth images, thermal infrared images, and near-infrared images to improve the performance of computer vision tasks have attracted more and more attention. Thermal infrared images reflect the temperature field of the surface of an object. Their imaging is insensitive to light and has a strong ability to penetrate haze and smoke. Therefore, compared with visible light images, infrared thermal images have the characteristics of strong anti-interference ability, not easily affected by bad weather and visible light, and strong target recognition ability. If the target is not well imaged under visible light, thermal infrared images can be used to further assist in detection, which can overcome the effects of poor lighting, cluttered backgrounds, and bad weather to a certain extent. However, under certain complex imaging conditions, it is easy to produce situations where the contrast is not high enough, the edges are blurred, the target is incomplete or even lost, and at this time, visible light imaging provides detailed appearance and texture information, which provides useful information for salient target detection. The visible light-thermal infrared salient target detection task was born, which focuses on segmenting the common salient areas of paired visible light images and thermal infrared images.

[0004] Existing visible light-thermal infrared salient object detection methods can be divided into two categories: one is based on traditional multimodal salient object detection methods, and the other is based on deep learning. Traditional visible light-thermal infrared salient object detection methods complete salient detection through manually extracted features such as color, direction, texture, etc., which are overly dependent on manually selected features, have poor generalization ability for scenes, and cannot model high-level context features. In recent years, with the development of deep learning, especially the emergence of convolutional neural networks, the performance of salient object detection has been greatly improved. The latest visible light-thermal infrared salient object detection is based on a convolutional neural network to explore the cross-modal complementarity of two modal data fusion networks, and designs different methods to fuse multimodal, multi-level, and multi-scale features. These methods have achieved better results in terms of visible light-thermal infrared modal feature complementarity.

[0005] It is not difficult to see from the experimental results of multimodal salient object detection that there are two kinds of erroneous detection results, namely false detection and missed detection, and reducing false detection and missed detection is reflected in the evaluation indicators as improving precision and recall. High-quality salient object detection results need to maximize the recall and precision of the salient object area and finally achieve a balance between the two. Most of the previous multimodal salient object detection methods designed complex cross-modal networks for feature extraction, fusion and object detection, but ignored the characteristics of the two detection errors, false detection and missed detection, so no one paid attention to how to simultaneously improve the precision and recall and achieve a balance. There are the following problems:

[0006] On the one hand, due to the focus on designing multimodal fusion methods, the models of existing visible light-thermal infrared salient object detection methods are too complex, resulting in higher research costs;

[0007] On the other hand, in traditional methods, improving the recall rate requires lowering the detection threshold, while improving the precision rate requires raising the detection threshold. In deep methods, improving the recall rate often requires the model to pay more attention to local features, while improving the precision rate requires the model to pay more attention to global features. Existing methods often use a single network model to minimize the overall detection error, so the model often only focuses on improving the recall rate or the precision rate, ignoring their respective characteristics, and failing to achieve a balance between the two. Therefore, we focus on the two detection errors of false detection and missed detection in multimodal salient object detection, and by simultaneously improving the recall rate and precision rate and achieving a balance between them, we can collaboratively guide the detection of salient objects, so that the final detection result is optimal. Summary of the invention

[0008] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and to provide a visible light-thermal infrared salient target detection method based on a conditional generative adversarial network. The present invention can make good use of the complementary features of the two modalities and better detect the salient target area without the need for a complex multimodal fusion method, thereby obtaining a salient image with better accuracy and higher confidence.

[0009] Technical solution: A visible light-thermal infrared salient target detection method based on a collaborative generative adversarial network of the present invention comprises the following steps:

[0010] Step S1, inputting a visible light image and a corresponding thermal infrared image of the same image, using two generators, each generator containing an encoder, a decoder and a collaborative attention module CAM; and each generator uses an encoder-decoder with shared parameters to process the features of the visible light modality and the thermal infrared modality, the encoder encodes to obtain multi-level features of the input image (i.e., the visible light modality and the thermal infrared modality), and marks these features as f1-f5;

[0011] Step S2: For the two generators, the deepest features f5 obtained by each generator are processed by the global context information module GCIM to obtain the global features f global Then, f1~f5 obtained in step S1 and the global feature f obtained in step S2 are global Processed by the decoder; then the collaborative attention module CAM is used to fuse the features of the two modalities;

[0012] Step S3: The two generators obtain the predicted saliency maps S1 and S2 respectively. Finally, the output features of the two generators are integrated and convolved to obtain the final saliency map S final ;

[0013] Step S4: Use a discriminator to send S1, S2 and the true value GT to the discriminator for classification, and assist the generator to generate a more refined saliency map;

[0014] Step S5: supervise the training of the network model using the binary cross entropy BCE loss function, the loss function (used to reduce the false detection rate and the missed detection rate) and the conditional generative adversarial network loss function through the annotated true value.

[0015] Furthermore, the feature encoders of each generator in step S1 adopt a VGG19 network with shared parameters, and the VGG19 network removes the last fully connected layer. The VGG19 networks of the two generators extract hierarchical features from the visible light modality and the thermal infrared modality, respectively. The features of each layer are represented as f1~f5, and the number of channels are 32, 64, 128, 512 and 512, respectively.

[0016] Furthermore, in order to better obtain global context information, the global context information module GCIM in step S2 can expand the receptive field.

[0017] f global =GCIM(f5);

[0018] The global context information module GCIM consists of three parallel branches. Each branch m (m∈1,2,3) consists of a serial convolution layer (convolution kernel∈3,5,7) and a dilation convolution layer (dilation rate∈3,5,7). The number of output channels of each branch is 256. Then, the original feature f5 is connected with the features of the three branches by residual connection, and the number of channels is reduced to 256 by 1*1 convolution operation, so as to obtain the global feature f global .

[0019] Furthermore, the decoder in step S2 processes features f1 to f5 and global features f global The specific process is:

[0020] F output =Conv(Fi +f global ), i=1

[0021] F i =Conv(F i+1 +f global +f i ), 1 <i<=4

[0022] F i =Conv(f global ), i=5

[0023] Among them, f i is the extracted features of different layers of the encoder, i represents the features of layers 1 to 5, and F i It is the decoder feature of the corresponding layer. Conv represents a layer of 3*3 convolution and ReLU activation function to achieve nonlinear activation. The i+1 layer feature is kept consistent with the i layer feature size through upsampling operation.

[0024] The above processing process is mainly because in the salient target detection task, high-level features provide abstract global context information, which is helpful for the positioning of foreground targets, while low-level features contain a lot of detail information, which is conducive to the refinement of boundaries. Both play an important role in the segmentation of salient targets, so it is necessary to integrate multi-level features.

[0025] After the above encoding and decoding is completed, the collaborative attention module is needed to fuse the F1 features of the visible light modality and the thermal infrared modality and highlight the common salient targets in different modalities. Specifically, the collaborative attention module mines the channel features of the modality itself and the interdependence between cross-modalities, and then further explores the relationship between cross-modal features through spatial attention.

[0026] Furthermore, in the process of using the collaborative attention module CAM to fuse the features of the two modalities in step S2, the channel attention is used to mine the features of the modality itself, and the channel attention of the two modalities is obtained respectively, specifically:

[0027] CA k =σ(M(F k (p,q)));

[0028] k represents the visible light modality RGB or the thermal infrared modality T, M represents the maximum pooling operation, and p and q refer to pixels;

[0029] Use channel attention to fuse two modal features:

[0030]

[0031] Represents the corresponding pixel multiplication; get the fusion feature f fuse, and then use spatial attention to further highlight the salient target area. The spatial attention is expressed as:

[0032] S k =σ(Conv(F fuse (p,q)))

[0033] The final output feature is expressed as:

[0034]

[0035] * represents a 3*3 convolution operation, and W is the corresponding convolution parameter.

[0036] The above-mentioned cross-modal collaborative attention module can model the correlation between channel features and spatial features between the visible light modality and the thermal infrared modality, and adaptively select foreground area features.

[0037] Furthermore, in step S3, the two generators respectively generate their respective fused features F out The convolution operation and sigmoid activation function are used to obtain the predicted saliency maps S1 and S2 of the two generators; then the output features of the two generators are integrated, and the final saliency prediction map S is obtained after a convolution layer and sigmoid activation function activation. final :

[0038] S final =σ(W*[F out1 ,F out2 ]);

[0039] [*] indicates the connection operation of the corresponding channel, * indicates the 3*3 convolution operation, and W is the corresponding convolution parameter.

[0040] Furthermore, the discriminator in step S4 comprises four convolution layers, three max-pooling layers and three fully connected layers, and the input of the discriminator is three saliency maps with original input, [S1, input], [S2, input] and [S, input], where S1 and S2 are the prediction results of the two generators, S is the true value map, and the input is the average value of the original visible light image and the thermal infrared image;

[0041] Finally, the output of the discriminator is three values, which are the discriminant scores for each input.

[0042] Furthermore, the loss function G in step S5 * for:

[0043]

[0044] Among them, L α(G, D) is the conditional generation adversarial network loss function, L sg (S t ,Y) is the generator loss function, L S (S final ,Y) is the significant loss function, L sc (S1,S2) is the consistency loss function; S t Represents the saliency maps S1 and S2 generated by different generators; α, β, γ, ω are weight parameters that control different losses;

[0045] The significant loss is the final prediction result S final The cross entropy loss between θ and the true value;

[0046] Adversarial loss L α (G, D) is used to balance the generation network and the discriminator network during the training phase, helping the two generators to generate better segmentation results;

[0047]

[0048] X input Represents the average value of the input RGB image and thermal infrared image, S t Including S1 and S2, representing the saliency maps predicted by the two generators, Y represents the true value, and D() represents the discriminator function.

[0049] Furthermore, different generator losses are used in the above two generators to improve precision and recall respectively. For example, in generator 1, the recall is improved by reducing the false negative (FN) of the prediction graph. In Generator 2, the precision is improved by reducing false positives (FP),

[0050] The two generator losses are as follows:

[0051]

[0052]

[0053] λ1 and λ2 are used to balance FN and FP and control the two generators to favor their respective tasks. t Including S1 and S2, j represents the jth image in the dataset, and N is the total number of images in the dataset;

[0054] Given the final saliency map and the real data Y={Y n |n=1,...,T}, where n represents the nth pixel and T is the total number of pixels;

[0055] The binary cross entropy BCE loss function is:

[0056]

[0057] The consistency loss makes the two generators produce more refined segmentation maps. Although the tasks of the two sub-networks are different, their final segmentation results should be as similar to the true value as possible. In addition, the predicted maps S1 and S2 of the two generators should also be similar in theory. However, the adversarial loss and the generator loss make the salient map S1 and the salient map S2 approach the true value in their own way, and there is a gap after training.

[0058] Therefore, the consistency loss L is used here sc To narrow the generation gap between saliency map S1 and saliency map S2;

[0059]

[0060] Among them, w, h, and c represent the width, height, and number of channels of the output image, respectively.

[0061] Beneficial effects:

[0062] Based on the false detection and missed detection in salient detection, the present invention decomposes the complex RGBT salient target detection into two subtasks, and designs two corresponding generators to reduce the false detection rate and missed detection rate respectively, thereby improving the precision and recall rates. With the assistance of conditional generative adversarial networks, the precision and recall rates are balanced to obtain a better refined salient map. In addition, the collaborative attention module of cross-modal feature fusion helps to explore the complementarity of visible light modality and thermal infrared modality, while suppressing the influence of feature noise of different modalities. Finally, the framework of the present invention is very flexible and universal, and its backbone and even the framework of the entire generator can be taken by other corresponding frameworks. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0064] Figure 2 It is a schematic diagram of the network model of the present invention;

[0065] Figure 3 It is a schematic diagram of the structure of the global context information module of the present invention;

[0066] Figure 4 It is a schematic diagram of visual comparison in the embodiment. DETAILED DESCRIPTION

[0067] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0068] like Figure 1As shown, the visible light-thermal infrared salient target detection method based on collaborative generative adversarial network of the present invention comprises the following steps:

[0069] Step (1): Given a visible light image and an infrared image corresponding to the image, two VGG19 networks with fully connected layers removed are used as encoders. Both VGG19 networks extract multi-layer features of RGB mode and T mode, and mark these image features as f1~f5.

[0070] Step (2), f5 obtained in step (1) is subjected to GCIM to obtain f global .f global The features f1 to f5 obtained by the encoder as high-level guidance information are integrated through the decoder, and then the features of the two modalities are fused using the collaborative attention module CAM. The two encoders have their own decoders and CAMs. The encoder, decoder, and CAM form a generator, with a total of two generators to complete the tasks of reducing the false detection rate and missed detection rate respectively.

[0071] Step (3), the two generators obtain the predicted saliency map S1 and saliency map S2 respectively, and finally integrate the output features of the two generators and obtain the final saliency map S through convolution. final .

[0072] Step (5): Use a discriminator to send S1, S2 and the true value GT to the discriminator for classification, and assist the generator to generate a more refined saliency map.

[0073] Step (6), supervise the training of the network model using the binary cross entropy BCE loss function, the loss function for reducing the false positive rate and the missed positive rate, and the conditional generative adversarial network loss function through the annotated true value.

[0074] Based on the two different tasks of improving the recall and precision of saliency maps, the present invention designs two generator networks based on conditional generative adversarial networks, so that the two networks can segment saliency maps closer to the true value from their own perspectives in the process of mutual competition and mutual assistance, thereby obtaining a more refined final saliency map after integrating the features of the two generators. In addition, the collaborative attention module of cross-modal feature fusion exploits the complementarity of the visible light modality and the thermal infrared modality, while suppressing the influence of feature noise of different modalities, so that the final saliency map can be more accurate and have higher confidence.

[0075] like Figure 2 As shown, the network model of this embodiment includes two parts: a generator and a discriminator. Different from the classic conditional generative adversarial network (cGAN), this embodiment uses two generators to complete different tasks and combines their results to predict the final saliency map.

[0076] Each generator converts a pair of RGBT images into a binary saliency segmentation map, with the goal of improving recall and precision respectively. Formally, this is represented as G1(I rgb,t )->S1 and G2(I rgb,t )->S2, where S1 and S2 represent the segmentation results predicted by two generators. G1 and G2 represent two different generators.

[0077] In the generator of the twin network, this embodiment uses a collaborative attention module to fuse different modal features and constrain redundant feature noise in channels and spaces; finally, the output features of the two generators are combined to predict the final segmentation map. Based on cGAN, the discriminator is used to classify the three inputs of S1, S2 and true value.

[0078] like Figure 3 As shown, the global context information module of this embodiment is specifically as follows: the highest-level feature f5 extracted by the encoder of the global context module input, first, in order to reduce the amount of calculation, the number of input feature channels is reduced to 256 by using 1*1 convolution; secondly, this embodiment designs 3 branches, each branch uses serial convolution operations and dilated convolution operations, and the convolution kernels (dilated convolution kernels) of the three branches are 3, 5, and 7 respectively, and the number of output feature channels of the three branches is 256; finally, it is connected to the original feature in the channel dimension through the residual connection, and the global context information of the feature is fully extracted through this module.

[0079] Example:

[0080] This embodiment uses the publicly available visible light-thermal infrared salient target detection datasets, VT821, VT1000, and VT5000.

[0081] Among them, VT821 contains 821 pairs of visible light and thermal infrared image pairs, VT1000 contains 1000 pairs of visible light and thermal infrared image pairs, and VT5000 is divided into training set and test set, each containing 2500 pairs of visible light and thermal infrared image pairs.

[0082] This implementation uses the VT5000 training set to train the network. This implementation uses random spatial affine transformation to process the input image pairs, and uses random horizontal flipping data enhancement to obtain diverse data. The network of this embodiment is trained for 100 rounds on a TitanXp GPU with a batch size of 4. The stochastic gradient descent optimization method is used for training, and the generator learning rate is 10 -4 , the discriminator learning rate is 10 -5 , the input image size is 256×256.

[0083] To facilitate quantitative evaluation, this embodiment adopts four widely used indicators.

[0084] (1) Mean absolute error (MAE). MAE is defined as:

[0085]

[0086] Where T is the total number of pixels and S is the predicted saliency map, and Y is the true value map.

[0087] (2) F-measure (Fm). It is defined as the weighted harmonic mean of precision and recall.

[0088] The F-measure formula is:

[0089]

[0090] where β 2 =0.3, which means more attention is paid to accuracy.

[0091] (3) S-measure (S m ). m The target-aware structural similarity (S0) and region-aware structural similarity (S0) between the predicted image and the true value label are calculated. r ).

[0092] S m As shown below:

[0093] S m =α·S0+(1-α)·S r

[0094] where α is set to 0.5.

[0095] (4), E-measure (Em). This measure is an enhanced alignment measure that jointly captures image-level statistics and pixel-level matching information.

[0096] Example:

[0097] Compare the technical solution of the present invention with other prior arts.

[0098] This embodiment compares the network of the technical solution of the present invention with other 11 methods.

[0099] The comparison methods include 4 existing visible light-depth salient object detection methods transplanted to the visible light-thermal infrared salient object detection problem: DMRA, S2MA, A2dele, BBSNet. In addition, 8 existing visible light-thermal infrared salient object detection methods are also included: MTMR, M3S-NIR, SGDL, ADF, SiamDecoder, APNet, ECFFNet, CSRNet.

[0100] All results were generated using code provided by the authors.

[0101] Quantitative comparison:

[0102] The specific comparative test results of this embodiment are shown in Table 1. This embodiment uses Em, Sm, Fm, MAE and wFm to evaluate the corresponding saliency map on three data sets. Among all the evaluation indicators, the method of the present invention achieves the best performance. Compared with the suboptimal CSRNet method, the performance of the Em, Fm, MAE and wFm indicators of the three data sets is improved by an average of 1.4%, 2.3%, 0.7% and 1.6%, respectively, and the Sm indicator is basically the same.

[0103] Table 1 Schematic diagram of comparison of measurement indicators

[0104]

[0105] Qualitative evaluation:

[0106] like Figure 4 As shown, the final saliency map of this implementation is compared with that of other technical solutions. This implementation selected 10 pairs of challenging samples. It can be found from the figure that the method of the present invention is more accurate in segmenting the salient area. When the quality of the visible light modality is poor, the present invention can better utilize the information of the thermal infrared modality, and the obtained saliency prediction map has better quality (columns 4 and 7). In the challenge of large targets, most methods will lose part of the object, but the present invention will not have this problem (columns 1 and 10). In the challenge of cluttered scenes, most methods are more susceptible to background noise, thereby segmenting part of the background area into the foreground target. The technical solution of the present invention can well avoid this type of problem (columns 3, 6, and 10).

[0107] It can be judged from the above embodiments that the present invention can obtain more refined significant prediction results; at the same time, the features integrated by the global context information module of the present invention contain richer global context information; the collaborative attention model of the present invention fuses the visible light modality and thermal infrared modality features while further highlighting the foreground target area and suppressing the influence of redundant background noise.

Claims

1. A visible light-thermal infrared salient target detection method based on conditional generative adversarial network, characterized by: The following steps are involved: Step S1: Input a visible light image and a corresponding thermal infrared image of the same target, use two generators, each generator contains an encoder, a decoder and a collaborative attention module CAM; and each generator uses an encoder-decoder with shared parameters to process the features of the visible light modality and the thermal infrared modality, the encoder encodes to obtain multi-level features of the two modalities of the input image, and marks these features as f1-f5; Step S2: For the two generators, the deepest features f5 obtained by each generator are processed by the global context information module GCIM to obtain the global features f global Then, f1~f5 obtained in step S1 and the global feature f obtained in step S2 are global Processed by the decoder; then the collaborative attention module CAM is used to fuse the features of the two modalities; Step S2: The decoder processes features f1~f5 and global features f global The specific process is: If i = 1, then F output =Conv(F i +f global ); If 1 < i ≤ 4, then F i = Conv(F i+1 + f global + f i ); If i = 5, then F i =Conv(f gl obal); Among them, f i is the extracted features of different layers of the encoder, i represents the features of layers 1 to 5, and F i It is the decoder feature of the corresponding layer. Conv represents a layer of 3*3 convolution and ReLU activation function to achieve nonlinear activation. The i+1 layer feature is kept consistent with the i layer feature size through upsampling operation. In step S2, when the collaborative attention module CAM is used to fuse the features of the two modalities, the channel attention is used to mine the features of the modality itself, and the channel attention of the two modalities is obtained respectively, specifically: CA k =σ(M(F k (p,q))); k represents the visible light modality RGB or the thermal infrared modality T, M represents the maximum pooling operation, and p and q refer to pixels; Use channel attention to fuse two modal features: Represents the corresponding pixel multiplication; get the fusion feature f fuse , and then use spatial attention to further highlight the salient target area. The spatial attention is expressed as: SA k =σ(Conv(F fuse (p,q))) The final output feature is expressed as: * represents a 3*3 convolution operation, and W is the corresponding convolution parameter; Step S3: The two generators obtain the predicted saliency map S1 and saliency map S2 respectively. Finally, the output features of the two generators are integrated and convolved to obtain the final saliency map S final ; Step S4: Use a discriminator to send S1, S2 and the true value GT to the discriminator for classification, and assist the generator to generate a more refined saliency map; Step S5, using the labeled true value to supervise the training of the network model using a binary cross entropy BCE loss function, a loss function designed to reduce the false detection rate and the missed detection rate, and a conditional generative adversarial network loss function; The loss function G in step S5 * for: Among them, L α (G, D) is the conditional generation adversarial network loss function, L sg (S t ,Y) is the generator loss function, L S (S final ,Y) is the significant loss function, L sc (S1,S2) is the consistency loss function; S t Represents the saliency maps S1 and S2 generated by different generators; α, β, γ, ω are weight parameters that control different losses; The significant loss is the final prediction result S final The cross entropy loss between θ and the true value; The adversarial loss L(G,D) is used to balance the generative network and the discriminative network during the training phase, helping the two generators to generate better segmentation results; X input Represents the average value of the input RGB image and thermal infrared image, S t Including S1 and S2, representing the saliency maps predicted by the two generators, Y represents the true value, and D() represents the discriminator function.

2. The visible light-thermal infrared salient target detection method based on conditional generative adversarial network according to claim 1 is characterized in that: The feature encoders of each generator in step S1 adopt a VGG19 network with shared parameters, and the VGG19 network removes the last fully connected layer. The VGG19 networks of the two generators extract hierarchical features from the visible light modality and the thermal infrared modality, respectively. The features of each layer are represented as f1~f5, and the number of channels are 32, 64, 128, 512 and 512, respectively.

3. The visible light-thermal infrared salient target detection method based on conditional generative adversarial network according to claim 1 is characterized by: In the step S2, the global context information module GCIM is used to integrate the highest-level feature f5 and expand the receptive field in order to better obtain the global context information, which is expressed as follows: f global =GCIM(f5); The global context information module GCIM consists of three parallel branches. Each branch m includes a serial convolution layer and a dilated convolution layer. The number of output channels of each branch is 256. Then, the original feature f5 is connected with the features of the three branches by residual connection, and the number of channels is reduced to 256 by 1*1 convolution operation, so as to obtain the global feature f global .

4. The visible light-thermal infrared salient target detection method based on conditional generative adversarial network according to claim 1 is characterized by: In step S3, the two generators respectively generate their respective fused features F out The convolution operation and sigmoid activation function are used to obtain the predicted saliency map S1 and saliency map S2 of the two generators; then the output features of the two generators are integrated, and the final saliency prediction map S is obtained after a convolution layer and sigmoid activation function activation. final : S final =σ(W*[F out1 ,F out2 ]); [*] indicates the connection operation of the corresponding channel, * indicates the 3*3 convolution operation, and W is the corresponding convolution parameter.

5. The visible light-thermal infrared salient target detection method based on conditional generative adversarial network according to claim 1, characterized in that: The discriminator in step S4 comprises four convolution layers, three max-pooling layers and three fully connected layers. The input of the discriminator is three saliency maps with original input, [S1, input], [S2, input] and [S, input], where S1 and S2 are the prediction results of the two generators, i.e., saliency maps, S is the true value map, and the input is the average value of the original visible light image and the thermal infrared image; Finally, the output of the discriminator is three values, which are the discriminant scores for each input.

6. The visible light-thermal infrared salient target detection method based on conditional generative adversarial network according to claim 1, characterized in that: The generator loss functions in the two generators are: False Negative False Positive S t Including S1 and S2, j represents the jth image in the dataset, and N is the total number of images in the dataset; λ1 and λ2 are used to balance FN and FP and control the two generators to be biased towards their respective tasks; Given the final saliency map and the real data Y={Y n |n=1,...,T}, where n represents the nth pixel and T is the total number of pixels; The binary cross entropy BCE loss function is: Finally, the consistency loss L is used sc To narrow the generation gap between saliency map S1 and saliency map S2; Among them, w, h, and c represent the width, height, and number of channels of the output image, respectively.

Images