A collaborative saliency detection method and system based on feature enhancement and contrast learning

Abstract

The application discloses a kind of based on feature reinforcement and contrast learning's collaborative saliency detection method and system, the method includes: a group of training images is input VGG16 encoder, and the features of different levels of image are extracted, wherein VGG16 encoder is according to the depth of convolution layer and feature is recorded as x1,x2,x3,x4,x5 from shallow to deep;Feature x5 is input to collaborative saliency detector GAM after residual convolution layer, and the common saliency object of a group of images is obtained, and recorded as collaborative saliency prototype proto;With the aid of collaborative saliency prototype proto and feature enhancement module MSE, feature x1,x2,x3,x4,x5 is enhanced, and enhanced feature f1,f2,f3,f4,f5 is obtained;f1,f2,f3,f4,f5 and proto are input decoder for decoding, and then detection network is obtained, and the collaborative saliency map S of a group of images is predicted using the detection network.The application adopts the method of foreground-background contrast learning, reduces the influence of background noise by learning the difference between foreground features and background features, and reduces the false alarm rate of collaborative saliency detection.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, specifically to a collaborative saliency detection method and system based on feature enhancement and contrastive learning. Background Technology

[0002] Cosine saliency detection identifies common salient objects in a set of images by studying the inherent relationships between them. Cosine saliency detection can filter out a large amount of invalid data from multiple images at once, significantly reducing the computational burden of subsequent image processing. It is widely used in computer vision tasks such as image segmentation, image retrieval, and object detection.

[0003] Recent co-saliency detection methods, such as GCoNet (Group collaborative learning for co-salient object detection), utilize attention mechanisms to enhance co-saliency features, while IcNet (Icnet: intra-saliency correlation network for co-saliency detection) represents the shared attributes of co-saliency objects by calculating feature similarity. These methods improve the accuracy of co-saliency detection, but two problems remain: first, they are easily influenced by other salient objects in the image (objects that are salient only in a single image, not co-saliency objects in a group of images); second, they are easily affected by background factors in complex scenes. Summary of the Invention

[0004] Purpose of the invention: In order to overcome the shortcomings of the prior art, the present invention provides a collaborative saliency detection method based on feature enhancement and contrastive learning. This method can solve the problems of low collaborative saliency detection rate and high false alarm rate in the prior art. The present invention also provides a collaborative saliency detection system based on feature enhancement and contrastive learning.

[0005] Technical solution: The collaborative saliency detection method based on feature enhancement and contrastive learning described in this invention includes:

[0006] The network construction and training process includes the following steps:

[0007] (1) Input a set of training images into the VGG16 encoder to extract features at different levels of the images. The VGG16 encoder records the features from shallow to deep as x1, x2, x3, x4, x5 according to the depth of the convolutional layer.

[0008] (2) After passing the feature x5 through the residual convolutional layer, it is input into the co-saliency detector GAM for encoding to obtain a set of common saliency objects of the images, which are denoted as co-saliency prototypes proto;

[0009] (3) Enhance features x1, x2, x3, x4, x5 with the help of the co-salience prototype proto and the feature enhancement module MSE to obtain enhanced features f1, f2, f3, f4, f5;

[0010] (4) Input f1, f2, f3, f4, f5 and proto into the decoder to obtain the detection network, and use the detection network to predict a co-saliency map S of a set of images;

[0011] The detection network test:

[0012] The test image set is input into the trained detection network to obtain the co-saliency map of the test image set.

[0013] Furthermore, including:

[0014] The loss function of the detection network includes: the IoU loss L between the co-saliency map S and the ground truth map Gt. IoU Loss L of the foreground-background contrast learning module fbc L fbc Defined as foreground feature map M+, background feature map M-, ground truth map Gt, and all-zero image Gt 0 Loss of focus between them;

[0015] Therefore, the loss function of the detection network is defined as:

[0016] L = L IoU +λL fbc

[0017] Where λ is the equilibrium L IoU Loss and L fbc The weighting coefficients of the loss.

[0018] Furthermore, including:

[0019] The foreground feature map M+ and background feature map M- are obtained through foreground-background contrast learning (FBC). In the foreground learning, the ground truth map Gt of the image is multiplied pixel by pixel with the feature map x5 and the co-salience prototype proto to obtain the foreground prototype F+, which is then passed through a CNN network to obtain the foreground feature map M+. In the background learning, the background mask 1-Gt of the image is multiplied pixel by pixel with the feature map x5 and the co-salience prototype proto to obtain the background prototype F-, which is then passed through a CNN network to obtain the background feature map M-.

[0020] Furthermore, including:

[0021] The calculation of the IoU loss between the collaborative saliency map S and the ground truth map Gt is as follows:

[0022]

[0023] Where S represents the collaborative saliency map predicted by the network, and Gt represents the ground truth map;

[0024] The loss L of the computational foreground-background contrast learning module fbc :

[0025]

[0026] Where, L FL It is the focus loss, where Gt is the true value. 0 For an all-zero image, <.> represents a join operation.

[0027] Furthermore, including:

[0028] In step (2), the method for generating the co-saliency prototype proto is as follows:

[0029] (21) The initial feature x5 is first processed by the residual block to generate the enhanced residual feature F. res :

[0030] F res =x5+conv 1x1 (x5),F res ∈R N×C×H×W

[0031] (22) F res Enter GAM, F res The feature maps K and Q are obtained by passing them through two 1×1 convolutional layers respectively.

[0032] K∈R N×C×H×W ,Q∈R N×C×H×W Reshape K and Q into K∈R NHW×C ,Q∈R NHW×C Calculate the feature similarity graph S of K and Q:

[0033] S = KQ T

[0034] Where S∈R NHW×NHW NHW represents the total number of pixels in N images in a set of images, and S represents the number of pixels p in feature map K. i With pixel p in feature map Q j The similarity values ​​between them, i = 1, 2, ..., NHW, j = 1, 2, ..., NHW;

[0035] (23) Reshape the feature similarity map S into S r ∈R NHW×N×HW Select the image with p in each image. i The value with the highest similarity is p. i The N maximum similarity values ​​are denoted as S. N-max ∈R NHW×N :

[0036]

[0037] (24) Take the N maximum similarity values ​​S N-max The average value is used as p i The significant probability of collaboration is used to generate a global collaborative attention graph P:

[0038]

[0039] (25) Normalize the global collaborative attention graph P using the softmax operation and reshape it into...

[0040] P m ∈R N×(1×H×W) Then P m It consists of N feature maps, each corresponding to a co-saliency attention map of N images in the input image group;

[0041] (26) The reshaped global collaborative attention graph P m With feature F res Multiplying them together, we get P. s ∈R N×(C×H×W) After passing through a 1×1 convolutional layer, the prototype of the co-saliency feature, proto, is obtained, where proto ∈ R. N×C×H×W .

[0042] Furthermore, including:

[0043] The Feature Enhancement Module (MSE) includes: a 1×1 convolutional layer, interpolation operations, addition operations, and a 3×3 convolutional layer. The feature enhancement process includes:

[0044] (31) Input the co-saliency prototype proto into a 1×1 convolutional layer so that its number of channels is the same as the number of channels of the feature map x1, and then perform interpolation operation to make the size of proto the same as the size of x1; add the adjusted co-saliency prototype proto and x1 pixel by pixel, and obtain the enhanced feature f1 through a 3×3 convolutional layer.

[0045] (32) Input the co-saliency prototype proto into a 1×1 convolutional layer to make its number of channels the same as the number of channels of feature map x2, and then interpolate to make the size of proto the same as the size of x2; add the adjusted co-saliency prototype proto and x2 pixel by pixel, and then pass through a 3×3 convolutional layer to obtain the enhanced feature f2.

[0046] (33) Input the co-saliency prototype proto into a 1×1 convolutional layer to make its number of channels the same as the number of channels of feature map x3, and then interpolate to make the size of proto the same as the size of x3; add the adjusted co-saliency prototype proto and x3 pixel by pixel, and then pass through a 3×3 convolutional layer to obtain the enhanced feature f3.

[0047] (34) Input the co-saliency prototype proto into a 1×1 convolutional layer to make its number of channels the same as the number of channels of feature map x4, and then interpolate to make the size of proto the same as the size of x4; add the adjusted co-saliency prototype proto and x4 pixel by pixel, and then pass through a 3×3 convolutional layer to obtain the enhanced feature f4.

[0048] (35) Input the co-saliency prototype proto into a 1×1 convolutional layer so that its number of channels is the same as the number of channels of feature map x5, and then interpolate so that the size of proto is the same as the size of x5; add the adjusted co-saliency prototype proto and x5 pixel by pixel, and then pass through a 3×3 convolutional layer to obtain the enhanced feature f5.

[0049] Furthermore, including:

[0050] In step (4), f1, f2, f3, f4, f5 and proto are decoded by the input decoder, including:

[0051] (41) Perform an interpolation operation on proto to make the size of proto the same as the size of f5, and then add it to f5 pixel by pixel to get p5.

[0052] (42) Interpolate p5 so that its size is the same as that of f4, and then add it to f4 pixel by pixel to get p4.

[0053] (43) Interpolate p4 so that its size is the same as that of f3, and then add it to f3 pixel by pixel to get p3.

[0054] (44) Interpolate p3 so that its size is the same as that of f2, and then add it pixel by pixel to f2 to get p2.

[0055] (45) Interpolate p2 so that its size is the same as that of f1, and then add it pixel by pixel to predict the co-saliency map S of a set of images.

[0056] On the other hand, the present invention provides a collaborative saliency detection system based on feature enhancement and contrastive learning, the system comprising:

[0057] The detection network construction module includes:

[0058] The feature extraction unit is used to input a set of training images into the VGG16 encoder and extract features at different levels of the images. The VGG16 encoder denotes the features from shallow to deep as x1, x2, x3, x4, x5 according to the depth of the convolutional layer.

[0059] The co-saliency prototype construction unit is used to encode the feature x5 after passing it through a residual convolutional layer into the co-saliency detector GAM, and obtain a set of common saliency objects of the images, denoted as co-saliency prototype proto;

[0060] The feature enhancement unit is used to enhance features x1, x2, x3, x4, x5 with the help of the co-salience prototype proto and the feature enhancement module MSE to obtain enhanced features f1, f2, f3, f4, f5;

[0061] The decoding unit is used to decode f1, f2, f3, f4, f5 and proto into the decoder to obtain the detection network, and use the detection network to predict a co-saliency map S of a set of images;

[0062] The detection network testing module is used to input the test image set into the trained detection network to obtain the co-saliency map of the test image set.

[0063] Based on the above, the present invention also provides an electronic device, the electronic device including a processor and a memory for storing processor-executable instructions; when the processor is configured to execute the instructions, the electronic device implements the method described above.

[0064] Based on the above, the present invention also provides a computer storage medium storing a computer program thereon, which, when executed by a computer processor, implements the method described above.

[0065] Beneficial effects: (1) The present invention uses the co-saliency prototype proto to enhance the co-saliency objects in the original features, which is beneficial to strengthen the common saliency features of a set of images, suppress objects that are only saliency in a single image, and improve the accuracy of co-saliency detection. (2) The present invention adopts the foreground-background contrast learning method, which reduces the influence of background noise by learning the differences between foreground features and background features, and reduces the false alarm rate of co-saliency detection. Attached Figure Description

[0066] Figure 1 This is a flowchart of the collaborative saliency detection method described in this invention;

[0067] Figure 2 This is a schematic diagram of the structure of the Collaborative Significant Prototype Generation Module (GAM) described in this invention;

[0068] Figure 3 This is a schematic diagram of the feature enhancement module (MSE) described in this invention;

[0069] Figure 4 This is a schematic diagram of the foreground-background contrast learning module (FBC) described in this invention;

[0070] Figure 5 This is a diagram showing the effect of the collaborative saliency detection described in this invention. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] First, the present invention provides a co-saliency detection method based on feature enhancement and contrastive learning, comprising: using a VGG convolutional neural network as an encoder to extract initial features at different levels of an image; inputting the deepest initial features into a co-saliency prototype generation module to obtain a co-saliency prototype; inputting the co-saliency prototype and the extracted initial features at different levels into a feature enhancement module to enhance the initial features; decoding the enhanced features and the co-saliency prototype together to obtain a co-saliency map; and enhancing the difference between foreground features and background regions through foreground-background contrastive learning during the training phase.

[0073] like Figure 1As shown, this invention employs a co-saliency detection method based on feature enhancement and contrastive learning. A VGG network is used as the encoder to extract features at different levels of the image. The resulting initial feature maps are denoted as x1, x2, x3, x4, and x5 according to their depth from shallow to deep. The initial feature x5 is input into the Co-saliency Prototype Generation (GAM) module to obtain the co-saliency prototype proto. Features x1, x2, x3, x4, and x5 are enhanced using the co-saliency prototype proto, resulting in enhanced features f1, f2, f3, f4, and f5. Finally, features f1, f2, f3, f4, f5, and proto are fed into the decoder to generate a co-saliency map. The ground truth map Gt, the background map (1-Gt), the initial feature x5, and the co-saliency prototype (proto) are input into the Foreground-Background Contrast (FBC) module to learn the differences between foreground features and background regions, reducing the false alarm rate of the detection results. The main detailed steps are as follows:

[0074] (1) Input multiple sets of training images into the VGG16 encoder and use the encoder to extract features at different levels of the images. Based on the depth of the convolutional layer, the features are recorded from shallow to deep as x1, x2, x3, x4, x5.

[0075] The specific method for feature extraction is as follows:

[0076] (11) Use the VGG16 network as an encoder to extract features at different levels of the image.

[0077] (12) The initial feature maps obtained are named x1, x2, x3, x4, x5 according to the depth from shallow to deep.

[0078] (2) After the feature x5 is passed through a 1×1 residual convolutional layer, it is input into the Co-salience Detector (GAM) for encoding to obtain a set of common salience objects of the images, denoted as co-salience prototype proto.

[0079] The specific method for generating the collaborative salient prototype is as follows:

[0080] (21) such as Figure 1 As shown, the initial feature x5 is first processed by the residual block to generate the enhanced residual feature F. res :

[0081] F res =x5+conv 1x1 (x5),F res ∈R N×C×H×W (4)

[0082] (22) such as Figure 2 As shown, F res Enter GAM, F resFeature maps K and Q (K∈R) are obtained by passing them through two 1×1 convolutional layers respectively. N×C×H×W ,Q∈R N×C×H×W Reshape K and Q into K∈R NHW×C ,Q∈R NHW×C Then, using formula (5), the feature similarity graph S of K and Q is calculated:

[0083] S = KQ T (5)

[0084] Where S∈R NHW×NHW NHW represents the total number of pixels in N images in a set of images, and S represents the number of pixels p in feature map K. i (i = 1, 2, ..., NHW) and pixel p in feature map Q j Similarity values ​​between (j = 1, 2, ..., NHW).

[0085] (23) such as Figure 2 As shown, the feature similarity map S is reshaped into S r ∈R NHW×N×HW Select the image with p in each image. i The value with the highest similarity is p. i The N maximum similarity values ​​are denoted as S. N-max ∈R NHW×N :

[0086]

[0087] (24) such as Figure 2 As shown, the N maximum similarity values ​​S N-max The average value is used as p i The significant probability of collaboration is used to generate a global collaborative attention graph P:

[0088]

[0089] (25) such as Figure 2 As shown, the global collaborative attention graph P is normalized using the softmax operation and reshaped into P0. m ∈R N×(1×H×W) Then P m It consists of N feature maps, each corresponding to a co-saliency attention map of N images in the input image group.

[0090] (26) such as Figure 2 As shown, the reshaped global collaborative attention graph P m With feature F res Multiplying them together, we get P. s ∈R N ×(C×H×W)After a 1×1 convolutional layer, the prototype of the co-saliency feature, proto, is obtained, where proto ∈ R. N×C×H×W .

[0091] (3) Use the co-salience prototype proto to enhance features x1, x2, x3, x4, x5 to obtain enhanced features f1, f2, f3, f4, f5.

[0092] The specific method of feature enhancement is as follows:

[0093] (31) such as Figure 3 As shown, the co-saliency prototype proto is input into a 1×1 convolutional layer so that its number of channels is the same as that of the feature map x1. Then, an interpolation operation is performed to make the size of proto the same as that of x1. The adjusted co-saliency prototype proto and x1 are added pixel by pixel and then passed through a 3×3 convolutional layer to obtain the enhanced feature f1.

[0094] (32) such as Figure 3 As shown, the co-saliency prototype proto is input into a 1×1 convolutional layer so that its number of channels is the same as that of feature map x2. Then, it is interpolated so that the size of proto is the same as that of x2. The adjusted co-saliency prototype proto and x2 are added pixel by pixel and then passed through a 3×3 convolutional layer to obtain the enhanced feature f2.

[0095] (33) such as Figure 3 As shown, the co-saliency prototype proto is input into a 1×1 convolutional layer so that its number of channels is the same as that of feature map x3. Then, it is interpolated so that the size of proto is the same as that of x3. The adjusted co-saliency prototype proto and x3 are added pixel by pixel and then passed through a 3×3 convolutional layer to obtain the enhanced feature f3.

[0096] (34) such as Figure 3 As shown, the co-saliency prototype proto is input into a 1×1 convolutional layer to make its number of channels the same as the number of channels in the feature map x4. Then, it is interpolated to make the size of proto the same as the size of x4. The adjusted co-saliency prototype proto and x4 are added pixel by pixel and then passed through a 3×3 convolutional layer to obtain the enhanced feature f4.

[0097] (35) such as Figure 3As shown, the co-saliency prototype proto is input into a 1×1 convolutional layer to make its number of channels the same as the number of channels of feature map x5, and then interpolated to make the size of proto the same as the size of x5. The adjusted co-saliency prototype proto and x5 are added pixel by pixel, and then passed through a 3×3 convolutional layer to obtain the enhanced feature f5.

[0098] (4) Input f1, f2, f3, f4, f5, and proto into the decoder to decode and predict a set of co-saliency maps S.

[0099] The specific method of decoding is as follows:

[0100] (41): For example Figure 1 As shown, proto is interpolated to make its size the same as f5, and then p5 is obtained by adding it pixel by pixel to f5.

[0101] (42): For example Figure 1 As shown, p5 is interpolated so that its size is the same as that of f4, and then p5 is added to f4 pixel by pixel to obtain p4.

[0102] (43): For example Figure 1 As shown, p4 is interpolated so that its size is the same as that of f3, and then p4 is added to f3 pixel by pixel to obtain p3.

[0103] (44): For example Figure 1 As shown, p3 is interpolated so that its size is the same as that of f2, and then p2 is obtained by adding it pixel by pixel to f2.

[0104] (45): Interpolate p2 so that its size is the same as that of f1, and then add it pixel by pixel to predict the co-saliency map S of a set of images.

[0105] (5) Foreground-background contrast learning (FBC): In foreground learning, the ground truth map Gt of the image is multiplied pixel by pixel with the feature map x5 and the co-salience prototype proto to obtain the foreground prototype F+, and then passed through a CNN network to obtain the foreground feature map M+; In background learning, the background mask 1-Gt of the image is multiplied pixel by pixel with the feature map x5 and the co-salience prototype proto to obtain the background prototype F-, and then passed through a CNN network to obtain the background feature map M-.

[0106] The specific method of foreground-background contrast learning is as follows:

[0107] (51) such as Figure 4 As shown, the ground truth map Gt of a set of images is multiplied pixel by pixel with the feature map x5 and the cosaliency prototype proto to obtain the foreground prototype F+; then it is passed through a CNN network consisting of 3×3 convolutions and 1×1 convolutions to obtain the foreground feature map M+.

[0108] (52) such as Figure 4 As shown, the background mask 1-Gt of a set of images is multiplied pixel by pixel with the feature map x5 and the cosaliency prototype proto to obtain the background prototype F-; then it is passed through a CNN network consisting of 3×3 convolution and 1×1 convolution to obtain the background feature map M-;

[0109] (6) The network's loss function consists of two parts: the IoU loss L between the co-saliency map S and the ground truth map Gt. IoU Loss L of the foreground-background contrast learning module fbc L fbc Defined as foreground feature map M+, background feature map M-, ground truth map Gt, and all-zero image Gt 0 The focus of the loss.

[0110] The final loss function is defined as:

[0111] L = L IoU +λL fbc (8)

[0112] Where λ is the equilibrium L IoU Loss and L fbc The weighting coefficients of the loss.

[0113] The loss function is:

[0114] (61) Calculate the IoU loss between the cosine saliency map S and the ground truth map Gt:

[0115]

[0116] Where S represents the collaborative saliency map predicted by the network, and Gt represents the ground truth map.

[0117] (62) Calculate the loss L of the foreground-background contrast learning module fbc L fbc Defined as foreground feature map M+, background feature map M-, ground truth map Gt, and all-zero image Gt 0 Loss of focus between:

[0118]

[0119] Where, L FL It is the focus loss, where Gt is the true value. 0For an all-zero image, <.> represents a join operation.

[0120] (63) Calculate the overall loss function of the network: L = L IoU +λL fbc

[0121] Where λ is the equilibrium L IoU Loss and L fbc The weighting coefficients of the loss.

[0122] (7) Input the test image group into the trained network to obtain the co-saliency map of the test image group; it should be noted that the foreground-background contrast learning in step 5 only works during the training phase and is ignored during the testing phase.

[0123] The following are experiments conducted according to the method of the present invention, illustrating the effects of the present invention, such as... Figure 5 As shown.

[0124] Experimental environment: PyTorch 1.8.1; Python 3.8; CUDA 11.1; GPU RTX2080ti.

[0125] Dataset: The training set used in the experiments was COCO9213, a subset of the COCO dataset containing 9213 images from 65 groups. Testing was conducted on Costal2015, a large dataset widely used for evaluating co-saliency detection.

[0126] Testing metrics: This invention uses the S-measure (a measure of structural similarity between a saliency map and its corresponding ground truth map) as the evaluation metric.

[0127] Table 1 shows a comparison of the scores of the method of this invention with those of GCAGC, CoEGNet, GICD, and GCoNet when trained on the Costal2015 dataset.

[0128] Table 1

[0129]

[0130] In summary, the S-measure value of this invention is higher than that of other models. This invention can effectively reduce the influence of background noise in images and suppress objects that are significant only in a single image, thereby improving the accuracy and precision of cosaliency detection.

[0131] On the other hand, the present invention also provides a collaborative saliency detection system based on feature enhancement and contrastive learning, comprising:

[0132] The detection network construction module includes:

[0133] The feature extraction unit is used to input a set of training images into the VGG16 encoder and extract features at different levels of the images. The VGG16 encoder denotes the features from shallow to deep as x1, x2, x3, x4, x5 according to the depth of the convolutional layer.

[0134] The co-saliency prototype construction unit is used to encode the feature x5 after passing it through a residual convolutional layer into the co-saliency detector GAM, and obtain a set of common saliency objects of the images, denoted as co-saliency prototype proto;

[0135] The feature enhancement unit is used to enhance features x1, x2, x3, x4, x5 with the help of the co-salience prototype proto and the feature enhancement module MSE to obtain enhanced features f1, f2, f3, f4, f5;

[0136] The decoding unit is used to decode f1, f2, f3, f4, f5 and proto into the decoder to obtain the detection network, and use the detection network to predict a co-saliency map S of a set of images;

[0137] The detection network testing module is used to input the test image set into the trained detection network to obtain the co-saliency map of the test image set.

[0138] Based on the above, the present invention also provides an electronic device, the electronic device including a processor and a memory for storing processor-executable instructions; when the processor is configured to execute the instructions, the electronic device implements the method described above.

[0139] Based on the above, the present invention also provides a computer storage medium storing a computer program thereon, which, when executed by a computer processor, implements the method described above.

[0140] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0144] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0145] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Therefore, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A collaborative saliency detection method based on feature enhancement and contrastive learning, characterized in that, include: The network construction and training process includes the following steps: (1) Input a set of training images into the VGG16 encoder to extract features at different levels of the images. The VGG16 encoder records the features from shallow to deep as x1, x2, x3, x4, x5 according to the depth of the convolutional layer. (2) After passing the feature x5 through the residual convolutional layer, it is input into the co-saliency detector GAM for encoding to obtain a set of common saliency objects of the images, which are denoted as co-saliency prototype proto; (3) Enhance features x1, x2, x3, x4, x5 with the help of the co-salience prototype proto and the feature enhancement module MSE to obtain enhanced features f1, f2, f3, f4, f5; (4) Input f1, f2, f3, f4, f5 and proto into the decoder to obtain the detection network, and use the detection network to predict a co-saliency map S of a set of images; The detection network test: The test image set is input into the trained detection network to obtain the co-saliency map of the test image set; The loss function of the detection network includes: the IoU loss between the collaborative saliency map S and the ground truth map Gt. Loss of the foreground-background contrast learning module ; Defined as foreground feature map M+, background feature map M-, ground truth map Gt, and all-zero image. Loss of focus between them; Therefore, the loss function of the detection network is defined as: ； in, It is balance Loss and The weighting coefficients of the loss; The foreground feature map M+ and background feature map M- are obtained through foreground-background contrast learning (FBC). In the foreground learning, the ground truth map Gt of the image is multiplied pixel-by-pixel with the feature map x5 and the co-saliency prototype proto to obtain the foreground prototype F+, which is then passed through a CNN network to obtain the foreground feature map M+. In the background learning, the background mask 1-Gt of the image is multiplied pixel-by-pixel with the feature map x5 and the co-saliency prototype proto to obtain the background prototype F-, which is then passed through a CNN network to obtain the background feature map M-. In step (2), the method for generating the co-saliency prototype proto is as follows: (21) The initial feature x5 is first processed by the residual block to generate enhanced residual features. : ； (22) will Enter GAM, The feature maps K and Q are obtained by passing them through two 1×1 convolutional layers respectively. , Reshape K and Q into , Calculate the feature similarity graph S between K and Q: ； in, NHW represents the total number of pixels in N images of a set of images, and S represents the number of pixels in feature map K. With pixels in feature map Q The similarity values ​​between them, i=1,2,...,NHW, j=1,2,...,NHW; (23) Reshape the feature similarity map S into Select the image with... The value with the highest similarity is obtained. The N maximum similarity values ​​are denoted as : ； (24) Take the N maximum similarity values The average value is used as The significant probability of collaboration is used to generate a global collaborative attention graph P: ， ； (25) Normalize the global collaborative attention graph P using the softmax operation and reshape it into ,but It consists of N feature maps, each corresponding to a co-saliency attention map of N images in the input image group; (26) The reshaped global collaborative attention diagram With features Multiply, we get After passing through a 1×1 convolutional layer, the prototype of the co-saliency feature, proto, is obtained. .

2. The collaborative saliency detection method based on feature enhancement and contrastive learning according to claim 1, characterized in that, Calculate the IoU loss between the co-saliency map S and the ground truth map Gt: ； Where S represents the collaborative saliency map predicted by the network, and Gt represents the ground truth map; Computational foreground-background contrast learning module loss : ； in, It is the focal loss, where Gt is the truth value. For images with all zeros, For connection operations.

3. The collaborative saliency detection method based on feature enhancement and contrastive learning according to claim 1, characterized in that, The Feature Enhancement Module (MSE) includes: a 1×1 convolutional layer, interpolation operations, addition operations, and a 3×3 convolutional layer. The feature enhancement process includes: (31) Input the co-saliency prototype proto into a 1×1 convolutional layer so that its number of channels is the same as the number of channels of feature map x1, and then perform interpolation operation to make the size of proto the same as the size of x1; add the adjusted co-saliency prototype proto and x1 pixel by pixel, and obtain the enhanced feature f1 through a 3×3 convolutional layer; (32) Input the co-saliency prototype proto into a 1×1 convolutional layer to make its number of channels the same as the number of channels of feature map x2, and then interpolate to make the size of proto the same as the size of x2; add the adjusted co-saliency prototype proto and x2 pixel by pixel, and then pass through a 3×3 convolutional layer to obtain the enhanced feature f2; (33) Input the co-saliency prototype proto into a 1×1 convolutional layer so that its number of channels is the same as that of the feature map x3, and then interpolate it so that the size of proto is the same as that of x3; add the adjusted co-saliency prototype proto and x3 pixel by pixel, and then pass it through a 3×3 convolutional layer to obtain the enhanced feature f3; (34) Input the co-saliency prototype proto into a 1×1 convolutional layer to make its number of channels the same as the number of channels of feature map x4, and then interpolate to make the size of proto the same as the size of x4; add the adjusted co-saliency prototype proto and x4 pixel by pixel, and then pass through a 3×3 convolutional layer to obtain the enhanced feature f4. (35) Input the co-saliency prototype proto into a 1×1 convolutional layer so that its number of channels is the same as that of the feature map x5, and then interpolate it so that the size of proto is the same as that of x5; add the adjusted co-saliency prototype proto and x5 pixel by pixel, and then pass it through a 3×3 convolutional layer to obtain the enhanced feature f5.

4. The collaborative saliency detection method based on feature enhancement and contrastive learning according to claim 1, characterized in that, In step (4), f1, f2, f3, f4, f5 and the proto input decoder are decoded, including: (41) Perform interpolation on proto to make the size of proto the same as the size of f5, and then add it to f5 pixel by pixel to get p5; (42) Interpolate p5 so that its size is the same as that of f4, and then add it pixel by pixel to f4 to get p4; (43) Interpolate p4 so that its size is the same as that of f3, and then add it pixel by pixel to f3 to get p3; (44) Interpolate p3 so that its size is the same as that of f2, and then add it pixel by pixel to f2 to get p2; (45) Interpolate p2 so that its size is the same as that of f1, and then add it pixel by pixel to predict a co-saliency map S of a set of images.

5. A collaborative saliency detection system based on feature enhancement and contrastive learning, using the collaborative saliency detection method based on feature enhancement and contrastive learning as described in claim 1, characterized in that, The system includes: The detection network construction module includes: The feature extraction unit is used to input a set of training images into the VGG16 encoder and extract features at different levels of the images. The VGG16 encoder denotes the features from shallow to deep as x1, x2, x3, x4, x5 according to the depth of the convolutional layer. The co-saliency prototype construction unit is used to encode the feature x5 after passing it through a residual convolutional layer into the co-saliency detector GAM, and obtain a set of common saliency objects of the images, denoted as co-saliency prototype proto. The feature enhancement unit is used to enhance features x1, x2, x3, x4, x5 with the help of the co-salience prototype proto and the feature enhancement module MSE to obtain enhanced features f1, f2, f3, f4, f5; The decoding unit is used to decode f1, f2, f3, f4, f5 and proto into the decoder to obtain the detection network, and use the detection network to predict a co-saliency map S of a set of images; The detection network testing module is used to input the test image set into the trained detection network to obtain the co-saliency map of the test image set.

6. An electronic device comprising a processor and a memory for storing processor-executable instructions; wherein the processor is configured to, when executing the instructions, cause the electronic device to perform the method as described in any one of claims 1 to 4.

7. A computer storage medium, characterized in that, It stores a computer program that, when executed by a computer processor, implements the method as described in any one of claims 1 to 4.

Images