Collaborative salient object detection method based on comprehensive feature mining and background noise reduction
By building a collaborative significant object detection network of comprehensive feature mining and background noise reduction, the problem of low prediction accuracy under the influence of background noise and occlusion in the prior art is solved, and higher detection accuracy and more accurate prediction significant graphs are achieved.
Patent Information
- Application Number
- CN202411073831.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-07
AI Technical Summary
The existing collaborative significant object detection method has low prediction accuracy under complex background noise and occlusion. The limitations of traditional attention mechanisms make feature detection difficult, resulting in feature misdetection and missed detection, and poor prediction quality.
The coordinated significant object detection method based on comprehensive feature mining and background noise reduction is adopted, and the detection network including backbone network, residual feature module, comprehensive feature generation module, feature enhancement module, decoder and adversarial learning module are constructed to perform feature extraction, Enhance and noise reduction to improve detection accuracy.
Improves the accuracy of coordinated significant object detection and the accuracy of predicting significant maps, reduces the deviation between prediction and source image labels, enhances the ability to suppress background noise, and can capture target details in complex situations.
Smart Images

Figure CN118628870B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of target detection, and in particular to a collaborative salient target detection method based on comprehensive feature mining and background noise reduction. Background Art
[0002] The application areas of collaborative salient object detection technology involve many aspects, such as image retrieval, video analysis, computer vision tasks, medical image processing, etc. At present, most collaborative salient object detection methods have the problems of low prediction accuracy and poor prediction quality, and the main reasons for the above problems are:
[0003] (1) Influence of background noise: At present, the background information of most images is relatively complex, resulting in a lot of background noise in the images. For example, some images contain a large number of non-cooperative salient targets, and the features of these non-cooperative salient targets are similar to those of cooperative salient targets. These situations interfere with the feature detection results of cooperative salient targets. In addition, there is also the problem that some cooperative salient targets are blocked. These situations will also interfere with the feature detection results of cooperative salient targets, easily causing false detection and missed detection of the features of cooperative salient targets. The feature detection of cooperative salient targets is difficult, and the prediction accuracy of the features of cooperative salient targets is low.
[0004] (2) Limitations of traditional attention mechanisms: In the field of co-salient object detection, most co-salient object detection methods directly use traditional attention mechanisms, which have their limitations. For example, traditional attention mechanisms tend to focus on a limited number of local related pixels. This causes the above-mentioned co-salient object detection methods to focus on a certain point of the co-salient object when performing co-salient object detection, and fail to pay attention to the co-salient object from a global perspective, which also causes the problem of poor prediction quality and low prediction accuracy of co-salient objects. Summary of the invention
[0005] In order to make up for the deficiencies of the prior art, the present invention provides a collaborative salient object detection method based on comprehensive feature mining and background noise reduction.
[0006] The technical solution of the present invention is:
[0007] A collaborative salient object detection method based on comprehensive feature mining and background noise reduction comprises the following steps:
[0008] S1, obtain the total training set;
[0009] S2. Construct a collaborative salient target detection network based on comprehensive feature mining and background noise reduction; the collaborative salient target detection network based on comprehensive feature mining and background noise reduction includes an Input layer, a backbone network, a residual feature module, a comprehensive feature generation module I, a feature enhancement module, a decoder and an adversarial learning module connected in sequence; the backbone network includes a feature extraction module I, a feature extraction module II, a feature extraction module III and a feature extraction module IV connected in sequence, and the output ends of the feature extraction module I, the feature extraction module II and the feature extraction module III are all connected to the input end of the decoder; the output end of the backbone network is also connected to the contrast learning module, and the output end of the residual feature module is also connected to the feature enhancement module; Inpu The output end of the t layer is also connected to the input end of the adversarial learning module; the backbone network is used to extract features from the image output by the Input layer to obtain feature maps of four different scales and containing rich contextual semantic information; the residual feature module is used to process the feature map output by the feature extraction module IV in the backbone network to obtain a feature map with residual features; the comprehensive feature generation module I performs feature enhancement and feature fusion on the features of the collaborative salient targets in the feature map output by the residual feature module, and reduces the background noise in the feature map output by the residual feature module, to obtain an average response map containing preliminary regional feature information of the collaborative salient targets and a comprehensive map containing rich information of the collaborative salient targets and low background noise. The feature enhancement module is used to perform feature fusion on the feature map output by the residual feature module, the average response map output by the comprehensive feature generation module I containing preliminary regional feature information of the collaborative salient target, and the comprehensive features that aggregate rich information of the collaborative salient target and have low background noise, and to perform feature enhancement on the features of the collaborative salient target, so as to obtain a feature map containing enhanced features of the collaborative salient target and more regional feature information of the collaborative salient target; the decoder is used to upsample the feature maps output by the feature extraction modules I, II, and III in the backbone network, and the feature maps output by the feature enhancement module, and to perform feature fusion on the features of the collaborative salient target in the feature map, so as to obtain to the predicted saliency map; the contrastive learning module is used to obtain the feature map and background feature map of the area containing only the collaborative salient target, and process the feature map and background feature map of the area containing only the collaborative salient target to obtain the collaborative saliency comprehensive feature and the background comprehensive feature; the collaborative saliency comprehensive feature and the background comprehensive feature and the comprehensive feature output by the comprehensive feature generation module I can be used to calculate the cosine similarity loss; the adversarial learning module multiplies the source image label and the source image pixel by pixel to obtain the image Ggt; and multiplies the predicted saliency map output by the decoder and the source image pixel by pixel to obtain the image Gm; and then predicts the probability value of the image Gm being the image Ggt and the probability value of the image Ggt being the image Ggt;
[0010] S3. Constructing the total generation loss of a collaborative salient object detection network based on comprehensive feature mining and background denoising and the total adversarial loss ;
[0011] S4. Obtain a sub-training set based on the total training set, and generate a loss in the total and the total adversarial loss Under the guidance of , the sub-training set is used to train the collaborative salient target detection network based on comprehensive feature mining and background noise reduction, and the collaborative salient target detection network model based on comprehensive feature mining and background noise reduction is obtained;
[0012] S5. Input the image to be detected into the trained collaborative salient object detection network model based on comprehensive feature mining and background noise reduction, and forward propagate once to obtain the predicted salient map and predicted probability value.
[0013] Preferably, step S1 specifically includes the following specific steps:
[0014] S1-1, obtain the original training set;
[0015] S1-2, data enhancement is performed on the images in the original training set to obtain the total training set, which specifically includes the following steps:
[0016] (1) Flip the images in the original training set left and right, with the probability of each image being flipped being 50%, to obtain the initial enhanced data set;
[0017] (2) Flip the images in the first enhanced data set upside down, with the probability of each image being flipped being 50%, to obtain the second enhanced data set;
[0018] (3) Perform color enhancement on all images in the secondary enhancement data set to obtain a total training set; wherein the color enhancement includes brightness enhancement, contrast enhancement, color enhancement and sharpness enhancement.
[0019] Preferably, in step S2, the residual feature module includes a convolution layer and an Add layer with a convolution kernel size of 1×1 connected in sequence, and the input end of the convolution layer with a convolution kernel size of 1×1 is also connected to the input end of the Add layer; wherein, the convolution layer with a convolution kernel size of 1×1 in the residual feature module is used to perform a convolution operation on the feature map output by the feature extraction module IV; the Add layer is used to perform a pixel-by-pixel addition operation on the feature map output by the feature extraction module IV and the feature map output by the convolution layer to obtain a feature map with residual features.
[0020] Preferably, in step S2, the comprehensive feature generation module I includes two parallel convolution layers with a convolution kernel size of 1×1, the input ends of the two convolution layers with a convolution kernel size of 1×1 are both connected to the output end of the Add layer of the residual feature module, and the output ends of the two convolution layers with a convolution kernel size of 1×1 are both connected to the matrix multiplication module, and the matrix multiplication module is sequentially connected to the seed selection module, the response module, and the element-by-element multiplication module; wherein, the input end of the seed selection module is connected to the output end of the Add layer of the residual feature module, and the input end of the element-by-element multiplication module is connected to the output end of the Add layer of the residual feature module.
[0021] Preferably, in step S2, the feature enhancement module includes three element-by-element multiplication modules, two matrix multiplication modules and two Add layers, wherein the first element-by-element multiplication module is connected to the output end of the element-by-element multiplication module in the comprehensive feature generation module I and the output end of the Add layer in the residual feature module, and the second element-by-element multiplication module is connected to the output end of the Add layer in the residual feature module and the output end of the response module in the comprehensive feature generation module I; the output ends of the first element-by-element multiplication module and the second element-by-element multiplication module are both connected to the first Add layer, and the first Add layer is sequentially connected to the MCA attention module, the convolution layer with a convolution kernel size of 1×1 and the ReLU activation layer, and the ReLU activation layer is divided into The Key convolution layer, Query convolution layer and Value convolution layer are respectively connected. The Key convolution layer and the Query convolution layer are both connected to the first matrix multiplication module. The first matrix multiplication module is respectively connected to the descending sorting module and the Softmax layer. The output ends of the descending sorting module and the Softmax layer are both connected to the input end of the third element-by-element multiplication module. The output end of the third element-by-element multiplication module and the output end of the Value convolution layer are both connected to the input end of the second matrix multiplication module. The output end of the second matrix multiplication module and the output end of the ReLU activation layer are both connected to the input end of the second Add layer. The output end of the second Add layer is connected to the input end of the decoder.
[0022] Preferably, in step S2, the comparative learning module includes two element-by-element multiplication modules and a comprehensive feature generation module II connected to the output ends of the two element-by-element multiplication modules, the input ends of the two element-by-element multiplication modules are connected to the output end of the feature extraction module IV in the backbone network, and the input ends of the two element-by-element multiplication modules are also respectively connected to the Input layer; the structure of the comprehensive feature generation module II in the present application is the same as the network structure of the comprehensive feature generation module I, and the function is also the same.
[0023] Preferably, in step S2, the adversarial learning module includes an input layer, two element-by-element multiplication modules, and a discriminator. The input layer of the adversarial learning module is connected to one of the element-by-element multiplication modules, and the other element-by-element multiplication module is connected to the output end of the decoder. The two element-by-element multiplication modules are also connected to the output end of the input layer connected to the backbone network, and the output ends of the two element-by-element multiplication modules are connected to the discriminator; the discriminator includes four discriminant modules connected in sequence, a fully connected layer, and a Sigmoid layer, wherein the four discriminant modules have the same structure and the same function;
[0024] Preferably, in step S3, the total generation loss of the collaborative salient object detection network based on comprehensive feature mining and background noise reduction is Including IoU loss 、BCE loss , cosine similarity loss and the generator loss Total generation loss It is used to constrain the update of weight parameters and hyperparameters of the backbone network, residual feature module, comprehensive feature generation module I, comprehensive feature generation module II, feature enhancement module and decoder, and the parameter updates of comprehensive feature generation module I and comprehensive feature generation module II are consistent; total adversarial loss Used to constrain the update of the weight parameters and hyperparameters of the discriminator in the adversarial learning module.
[0025] Preferably, step S4 specifically includes the following steps:
[0026] S4-1, randomly selecting N images and their corresponding labels for each category of collaborative salient target from the total training set described in step S1, where N is a natural number, and then scaling the above images and labels to sizes of 256×256×3 and 256×256×1, respectively, to obtain a sub-training set; in the training process of the collaborative salient target detection network based on comprehensive feature mining and background denoising described in the present application, a sub-training set is input in each training epoch, and the sub-training sets input in different training epochs are all obtained by using step S4-1;
[0027] Step S4-2: Input the sub-training set described in step S4-1 into the collaborative salient target detection network based on comprehensive feature mining and background noise reduction described in this application, perform forward propagation, and calculate the total generation loss of the collaborative salient target detection network based on comprehensive feature mining and background noise reduction respectively and the total adversarial loss , in total generated losses and the total adversarial loss Back propagation is performed under the guidance of , and the weight parameters and hyper parameters of the collaborative salient target detection network based on comprehensive feature mining and background noise reduction are updated, among which the total generation loss The weight parameters and hyperparameters of the constrained backbone network, residual feature module, comprehensive feature generation module I, comprehensive feature generation module II, feature enhancement module and decoder are updated, and the parameter updates of comprehensive feature generation module I and comprehensive feature generation module II are kept consistent. The total adversarial loss The weight parameters and hyperparameters of the discriminator in the constrained adversarial learning module are updated. After 220 epochs, the training process of the collaborative salient object detection network based on comprehensive feature mining and background noise reduction is completed, and the collaborative salient object detection network model based on comprehensive feature mining and background noise reduction is obtained; among them, the hyperparameters include the learning rate and the number of iterations.
[0028] Preferably, in step S5, the predicted probability value refers to the probability value that image Gm is image Ggt and the probability value that image Ggt is image Ggt.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] The collaborative salient target detection network based on comprehensive feature mining and background noise reduction constructed in the present application is more outstanding in capturing the details of collaborative salient targets, has higher detection accuracy, and the predicted salient map obtained is more accurate. The deviation between the predicted salient map and the label corresponding to the source image is less. This is mainly attributed to: the comprehensive feature generation module I constructed in the present application can perform feature enhancement, feature fusion and reduce the background noise in the feature map output by the residual feature module to obtain the average response map containing preliminary regional feature information of the collaborative salient target and the comprehensive features that aggregate the rich information of the collaborative salient target and have low background noise; the feature enhancement module constructed in the present application can perform feature enhancement, feature fusion and reduce the background noise in the feature map output by the residual feature module to obtain the average response map containing preliminary regional feature information of the collaborative salient target and the comprehensive features that aggregate the rich information of the collaborative salient target and have low background noise. The lower comprehensive features are feature fused and the features of the collaborative salient targets are feature enhanced to obtain a feature map containing the enhanced features of the collaborative salient targets and more regional feature information of the collaborative salient targets; the contrast learning module constructed in the present application is used to obtain the feature map and the background feature map of the area containing only the collaborative salient targets, and the feature map and the background feature map of the area containing only the collaborative salient targets are processed to obtain the collaborative salient comprehensive features and the background comprehensive features; the collaborative salient comprehensive features and the background comprehensive features and the comprehensive features output by the comprehensive feature generation module I can be used to calculate the cosine similarity loss; in addition, the adversarial learning module constructed in the present application can multiply the source image label and the source image pixel by pixel to obtain the image Ggt; and multiply the predicted saliency map output by the decoder and the source image pixel by pixel to obtain the image Gm; and then predict the probability value of the image Gm being the image Ggt and the probability value of the image Ggt being the image Ggt. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 Schematic diagram of the network structure of the collaborative salient target detection network based on comprehensive feature mining and background noise reduction in this application;
[0032] Figure 2 This is a schematic diagram of the network structure of the comprehensive feature generation module I in this application;
[0033] Figure 3 This is a schematic diagram of the network structure of the feature enhancement module in this application;
[0034] Figure 4 Schematic diagram of the network structure of the adversarial learning module in this application. DETAILED DESCRIPTION
[0035] A collaborative salient object detection method based on comprehensive feature mining and background noise reduction comprises the following steps:
[0036] S1, obtaining a total training set; wherein step S1 specifically includes the following specific steps:
[0037] S1-1, obtain the original training set;
[0038] The original training set in this application consists of the COCO9k dataset and the DUTS_class dataset. The COCO9k dataset is an existing dataset, and its acquisition URL is: https: / / drive.google.com / file / d / 1r6tRcSlvH8bXhaZD2VtGmHDxsXFl1v4z / view. The COCO9k dataset contains 65 categories of collaborative salient targets and 9213 images with mask information.
[0039] The DUTS_class dataset is an existing dataset, which can be obtained from: https: / / pan.baidu.com / s / 1m7imyp5gzgsCpA0xZFOW7Q#list / path=%2F. The DUTS_class dataset contains 291 categories of collaborative salient targets and 8,250 images with mask information.
[0040] S1-2, data enhancement is performed on the images in the original training set to obtain the total training set, which specifically includes the following steps:
[0041] (1) Flip the images in the original training set left and right, with the probability of each image being flipped being 50%, to obtain the initial enhanced data set;
[0042] (2) Flip the images in the first enhanced data set upside down, with the probability of each image being flipped being 50%, to obtain the second enhanced data set;
[0043] (3) Perform color enhancement on all images in the secondary enhancement data set to obtain a total training set; wherein the color enhancement includes brightness enhancement, contrast enhancement, color enhancement and sharpness enhancement; brightness enhancement refers to multiplying the brightness of each pixel of the image in the secondary enhancement data set by a random brightness enhancement coefficient, and the brightness enhancement coefficient is a random value in the range of greater than or equal to 0.5 and less than or equal to 1.5;
[0044] Contrast enhancement refers to multiplying the image in the brightness enhancement data set by a contrast enhancement coefficient, and the contrast enhancement coefficient is a random value within the range of greater than or equal to 0.5 and less than or equal to 1.5;
[0045] Color enhancement refers to multiplying the RGB value of each pixel in the image in the contrast enhancement dataset by a random color enhancement coefficient, where the random color enhancement coefficient is a random value in the range of greater than or equal to 0.0 and less than or equal to 2.0;
[0046] Sharpness enhancement means randomly sharpening all images in the color enhancement dataset using the ImageEnhance.Sharpness().enhance() function in the ImageEnhance module in the PIL library, and the random sharpness enhancement coefficient is a random value in the range of greater than or equal to 0.0 and less than or equal to 3.0.
[0047] S2. Construct a collaborative salient object detection network based on comprehensive feature mining and background noise reduction;
[0048] The structure of the collaborative salient object detection network based on comprehensive feature mining and background noise reduction is as follows: Figure 1 As shown, it includes an Input layer, a backbone network, a residual feature module, a comprehensive feature generation module I, a feature enhancement module, a decoder and an adversarial learning module connected in sequence; the backbone network includes a feature extraction module I, a feature extraction module II, a feature extraction module III and a feature extraction module IV connected in sequence, and the output ends of the feature extraction module I, the feature extraction module II and the feature extraction module III are all connected to the input end of the decoder; the output end of the backbone network is also connected to the contrast learning module, and the output end of the residual feature module is also connected to the feature enhancement module; the output end of the Input layer is also connected to the input end of the adversarial learning module; Figure 1 middle, represents the element-wise multiplication module, Indicates the Add layer, Represents the matrix multiplication module;
[0049] The backbone network is used to extract features from the image output by the Input layer, and obtain feature maps of four different scales that contain rich contextual semantic information.
[0050] The residual feature module is used to process the feature map output by the feature extraction module IV in the backbone network to obtain a feature map with residual features; and transmit the feature map with residual features to the comprehensive feature generation module I and the feature enhancement module respectively;
[0051] The comprehensive feature generation module I performs feature enhancement and feature fusion on the features of the collaborative salient target in the feature map output by the residual feature module, and reduces the background noise in the feature map output by the residual feature module, so as to obtain an average response map containing preliminary regional feature information of the collaborative salient target and a comprehensive feature that aggregates rich information of the collaborative salient target and has low background noise;
[0052] The feature enhancement module is used to fuse the feature map output by the residual feature module, the average response map output by the comprehensive feature generation module I containing preliminary regional feature information of the collaborative salient target, and the comprehensive features that aggregate rich information of the collaborative salient target and have low background noise, and to enhance the features of the collaborative salient target, so as to obtain a feature map containing enhanced features of the collaborative salient target and more regional feature information of the collaborative salient target;
[0053] The decoder is used to upsample the feature maps output by the feature extraction modules I, II, and III in the backbone network and the feature maps output by the feature enhancement module, and to perform feature fusion operations on the features of the collaborative salient targets in the feature maps to obtain a predicted salient map.
[0054] The contrastive learning module is used to obtain the feature map and background feature map of the area containing only the co-salient target, and process the feature map and background feature map of the area containing only the co-salient target to obtain the co-salient comprehensive features and the background comprehensive features; and the co-salient comprehensive features, the background comprehensive features and the comprehensive features output by the comprehensive feature generation module I can be used to calculate the cosine similarity loss;
[0055] The adversarial learning module multiplies the source image label and the source image pixel by pixel to obtain the image Ggt; and multiplies the predicted saliency map output by the decoder and the source image pixel by pixel to obtain the image Gm; and then predicts the probability value of image Gm being image Ggt and the probability value of image Ggt being image Ggt.
[0056] The structure and function of the backbone network in this application:
[0057] In this application, the network structure of the backbone network is as follows: Figure 1 As shown, the backbone network includes a feature extraction module I, a feature extraction module II, a feature extraction module III and a feature extraction module IV which are connected in sequence. The structures and functions of the feature extraction module I, the feature extraction module II, the feature extraction module III and the feature extraction module IV are respectively the same as the modules shown in stages 1 to 4 disclosed in the paper "PVT v2: Improved Baselines with Pyramid Vision Transformer".
[0058] In the present application, the feature extraction module I of the backbone network is used to extract features from the image output by the Input layer, and obtain a feature map with a size of 64×64×64 and containing contextual semantic information; the feature extraction module II is used to perform feature extraction operations on the feature map output by the feature extraction module I, and obtain a feature map with a size of 32×32×128 and containing contextual semantic information; the feature extraction module III is used to perform feature extraction operations on the feature map output by the feature extraction module II, and obtain a feature map with a size of 16×16×320 and containing contextual semantic information; the feature extraction module IV is used to perform feature extraction operations on the feature map output by the feature extraction module III, and obtain a feature map with a size of 8×8×512 and containing contextual semantic information. The backbone network of the present application has achieved the purpose of gradually aggregating higher-level semantic information through the gradual feature extraction from the feature extraction module I to the feature extraction module IV, and the feature map output by the feature extraction module IV contains richer contextual semantic information than the feature map output by the feature extraction module I, and has a larger receptive field. Since the feature extraction module I, feature extraction module II, feature extraction module III and feature extraction module IV used in this application respectively use the modules shown in stages 1 to 4 disclosed in the paper "PVT v2: Improved Baselines with Pyramid Vision Transformer", and the modules shown in stages 1 to 4 all use the self-attention mechanism, therefore, the feature extraction modules I to feature extraction modules IV used in this application can capture the collaborative salient features in the image on a global scale.
[0059] The structure and function of the residual feature module in this application:
[0060] The residual feature module in the present application includes a convolution layer and an Add layer with a convolution kernel size of 1×1 connected in sequence, and the input end of the convolution layer with a convolution kernel size of 1×1 is also connected to the input end of the Add layer; wherein, the convolution layer with a convolution kernel size of 1×1 in the residual feature module is used to perform a convolution operation on the feature map output by the feature extraction module IV; the Add layer is used to perform a pixel-by-pixel addition operation on the feature map output by the feature extraction module IV and the feature map output by the convolution layer to obtain a feature map with residual features. The setting of the residual feature module in the present application can effectively reduce the problem of gradient disappearance or gradient explosion.
[0061] The structure and function of the comprehensive feature generation module I in this application are as follows:
[0062] In this application, the network structure of the comprehensive feature generation module I is as follows: Figure 2As shown, it includes two parallel convolution layers with a convolution kernel size of 1×1, the input ends of the two convolution layers with a convolution kernel size of 1×1 are connected to the output ends of the Add layer of the residual feature module, and the output ends of the two convolution layers with a convolution kernel size of 1×1 are connected to the matrix multiplication module, and the matrix multiplication module is sequentially connected to the seed selection module, the response module, and the element-by-element multiplication module; wherein the input end of the seed selection module is connected to the output end of the Add layer of the residual feature module, and the input end of the element-by-element multiplication module is connected to the output end of the Add layer of the residual feature module; in this application, the seed selection module (SSB) has the same network structure and function as the Seed Selection Block (SSB) module disclosed in the paper "Democracy Does Matter: Comprehensive Feature Mining for Co-Salient Object Detection"; the response module (DRB) has the same network structure and function as the Democratic Response Block (DRB) module disclosed in the paper "Democracy Does Matter: Comprehensive Feature Mining for Co-Salient Object Detection";
[0063] In the present application, the functions of the comprehensive feature generation module I are as follows: the two convolution layers with a convolution kernel size of 1×1 in the comprehensive feature generation module I are both used to perform convolution operations on the feature maps output by the Add layer in the residual feature module; the matrix multiplication module is used to perform matrix multiplication operations on the feature maps output by the two convolution layers to obtain a feature similarity mapping matrix; the seed selection module is used to perform seed feature selection operations on the feature maps output by the Add layer in the residual feature module and the feature similarity mapping matrix output by the matrix multiplication module to achieve preliminary enhancement of the features of the collaborative salient targets and obtain a feature map containing the basic features of the collaborative salient targets in the image; the response module is used to select the seed features of the seed selection module The output feature map is subjected to response map convolution operation to achieve further preliminary enhancement of the features of the collaborative salient target, and an average response map containing preliminary regional feature information of the collaborative salient target is obtained; the element-by-element multiplication module is used to perform element-by-element multiplication operation on the feature map output by the Add layer in the residual feature module and the average response map output by the response module to achieve preliminary feature fusion of the features of the collaborative salient target; then, the torch.mean() function is used to perform an average operation on the feature map output by the element-by-element multiplication module to achieve further feature fusion of the features of the collaborative salient target, and obtain a comprehensive feature that aggregates rich information of the collaborative salient target and has low background noise.
[0064] The comprehensive feature generation module I constructed in the present application can effectively reduce the background noise in the source image. This enables the collaborative salient target detection network based on comprehensive feature mining and background noise reduction described in the present application to better capture the details of the collaborative salient targets in situations with complex backgrounds, many targets, more occlusions and other interference information, and can have higher detection accuracy.
[0065] The structure and function of the feature enhancement module in this application:
[0066] The network structure of the feature enhancement module in this application is as follows: Figure 3 As shown in FIG. 1 , the feature enhancement module includes three element-by-element multiplication modules, two matrix multiplication modules and two Add layers, wherein the first element-by-element multiplication module is connected to the output of the element-by-element multiplication module in the comprehensive feature generation module I and the output of the Add layer in the residual feature module, and the second element-by-element multiplication module is connected to the output of the Add layer in the residual feature module and the output of the response module in the comprehensive feature generation module I; the outputs of the first element-by-element multiplication module and the second element-by-element multiplication module are both connected to the first Add layer, and the first Add layer is sequentially connected to the MCA attention module, the convolution layer with a convolution kernel size of 1×1 and the ReLU activation layer, and the ReLU activation layer is respectively connected to Key convolution layer, Query convolution layer and Value convolution layer, Key convolution layer and Query convolution layer are connected to the first matrix multiplication module, the first matrix multiplication module is connected to the descending sorting module and the Softmax layer respectively, the output of the descending sorting module and the Softmax layer are connected to the input of the third element-by-element multiplication module, the output of the third element-by-element multiplication module and the output of the Value convolution layer are connected to the input of the second matrix multiplication module, the output of the second matrix multiplication module and the output of the ReLU activation layer are connected to the input of the second Add layer, and the output of the second Add layer is connected to the input of the decoder. Among them, the MCA attention module used in this application has the same structure and function as the Multidimensional CollaborativeAttention Module (MCA) disclosed in the paper "MCA: Multidimensional collaborative attention in deep convolutional neural networks for image recognition".
[0067] The functions of the feature enhancement module in this application are:
[0068] The first element-by-element multiplication module in the feature enhancement module is used to perform element-by-element multiplication on the feature map output by the element-by-element multiplication module in the comprehensive feature generation module I and the feature map output by the Add layer in the residual feature module; the second element-by-element multiplication module is used to perform element-by-element multiplication on the feature map output by the Add layer in the residual feature module and the feature map output by the response module in the comprehensive feature generation module I; the first Add layer is used to perform pixel-by-pixel addition on the feature map output by the first element-by-element multiplication module and the feature map output by the second element-by-element multiplication module to obtain a fused feature map containing comprehensive feature information and rich information of collaborative salient targets. The first element-by-element multiplication module, the second element-by-element multiplication module and the first Add layer in the feature enhancement module of this application are used to perform element-by-element multiplication on the feature map output by the element-by-element multiplication module. The setting of the layers can further realize the feature fusion of the features of the collaborative salient targets; the MCA attention module is used to capture the long-distance dependencies between the features in the fused feature map output by the first Add layer from the three dimensions of spatial dimension W, spatial dimension H and channel dimension C, so as to further enhance the features of the collaborative salient targets in the feature map and improve the accuracy of locating the collaborative salient targets; the convolution layer with a convolution kernel size of 1×1 is used to perform convolution operations on the feature map output by the MCA attention module; the ReLU activation layer is used to activate the feature map output by the convolution layer; the Key convolution layer is used to perform convolution operations on the feature map output by the ReLU activation layer to obtain the Key feature map; the Query convolution layer is used to perform convolution operations on the feature map output by the ReLU activation layer to obtain the Key feature map; The feature map output by the activation layer is convolved to obtain the Query feature map; the Value convolution layer is used to convolve the feature map output by the ReLU activation layer to obtain the Value feature map; the first matrix multiplication module is used to perform matrix multiplication on the Key feature map output by the Key convolution layer and the Query feature map output by the Query convolution layer to obtain the initial attention map; the Softmax layer is used to normalize the initial attention map output by the first matrix multiplication module to obtain the normalized attention map; the descending sorting module is used to sort each value of the initial attention map output by the first matrix multiplication module in descending order to obtain the sorting index matrix, and use the sorting index matrix to calculate the adjusted attention weight to obtain the adjusted attention weight. The third element-by-element multiplication module is used to perform element-by-element multiplication on the adjusted attention weight matrix output by the descending sorting module and the normalized attention map output by the Softmax layer to obtain the final attention map, which contains more regional feature information of the collaborative salient targets; the second matrix multiplication module is used to perform matrix multiplication on the Value feature map output by the Value convolution layer and the feature map output by the third element-by-element multiplication module; the second Add layer is used to perform element-by-element addition on the feature map output by the ReLU activation layer and the feature map output by the second matrix multiplication module to obtain a feature map containing enhanced features of the collaborative salient targets and more regional feature information of the collaborative salient targets.
[0069] In this application, the calculation method for adjusting the attention weight is calculated using the sorting index matrix, as shown in formula (1):
[0070] (1)
[0071] In formula (1), is the adjusted attention weight, is the magnification factor, is a sorting index matrix, in this application is one of the parameters iteratively updated during the training process of the collaborative salient target detection network based on comprehensive feature mining and background noise reduction. During the training process, the collaborative salient target detection network based on comprehensive feature mining and background noise reduction iteratively updates itself. i is the sorting index matrix The number of rows, j is the sorting index matrix The number of columns.
[0072] The structure and function of the comparative learning module in this application:
[0073] The network structure of the comparison learning module in this application is as follows: Figure 1 As shown, the contrastive learning module includes two element-by-element multiplication modules and a comprehensive feature generation module II connected to the output ends of the two element-by-element multiplication modules. The input ends of the two element-by-element multiplication modules are connected to the output end of the feature extraction module IV in the backbone network, and the input ends of the two element-by-element multiplication modules are also connected to the Input layer respectively. The structure of the comprehensive feature generation module II in the present application is the same as the network structure of the comprehensive feature generation module I, and the function is also the same.
[0074] Function of the contrastive learning module: In the contrastive learning module, one element-by-element multiplication module is used to perform element-by-element multiplication on the feature map output by the feature extraction module IV in the backbone network and the label of the source image input through the input layer to obtain a feature map of the area containing only the co-salient targets; the other element-by-element multiplication module is used to perform element-by-element multiplication on the feature map output by the feature extraction module IV in the backbone network and the inverted label to obtain a background feature map, where the inverted label refers to the label obtained by inverting the label of the source image; the feature maps output by the two element-by-element multiplication modules are processed by the comprehensive feature generation module II to obtain the co-salient comprehensive features and the background comprehensive features, respectively.
[0075] The structure and function of the adversarial learning module in this application:
[0076] The network structure of the adversarial learning module in this application is as follows: Figure 4As shown, it includes an Input layer, two element-by-element multiplication modules, and a discriminator. The Input layer of the adversarial learning module is connected to one of the element-by-element multiplication modules, and the other element-by-element multiplication module is connected to the output end of the decoder. The two element-by-element multiplication modules are also connected to the output end of the Input layer connected to the backbone network, and the output ends of the two element-by-element multiplication modules are connected to the discriminator; the discriminator includes four discriminant modules connected in sequence and a fully connected layer and a Sigmoid layer, wherein the four discriminant modules have the same structure and function; the discriminant module in this application has the same structure and function as the discriminant module disclosed in the URL https: / / github.com / ZhengPeng7 / MCCL / blob / main / adv.py.
[0077] Functions of the adversarial learning module: One element-by-element multiplication module is used to multiply the source image label input through the Input layer in the adversarial learning module and the source image input through the Input layer connected to the backbone network pixel by pixel to obtain the image Ggt; another element-by-element multiplication module is used to multiply the predicted saliency map output by the decoder and the source image input through the Input layer connected to the backbone network pixel by pixel to obtain the image Gm; the discriminator is used to predict the probability value of image Gm being image Ggt and the probability value of image Ggt being image Ggt.
[0078] The feature map output by the fourth discrimination module of the present application is processed by the fully connected layer and the Sigmoid activation layer in sequence to obtain the probability value of the predicted image Gm being the image Ggt and the probability value of the image Ggt being the image Ggt.
[0079] S3. Constructing the total generation loss of a collaborative salient object detection network based on comprehensive feature mining and background denoising and the total adversarial loss ;
[0080] The total generation loss of the collaborative salient object detection network based on comprehensive feature mining and background noise reduction in this application Including IoU loss 、BCE loss , cosine similarity loss and the generator loss , total generation loss , IoU loss 、BCE loss , cosine similarity loss and the generator loss The relationship between them is shown in formula (2):
[0081] (2)
[0082] In formula (2), , , and All are hyperparameters. , , and are set to 1, 0.1, 0.1, 1, and 1 respectively.
[0083] In formula (2), IoU loss The calculation formula is shown in formula (3):
[0084] (3)
[0085] In formula (3), represents the label of the source image, Represents the encoder pair label The predicted saliency map obtained by predicting the corresponding source image, N Represents the number of labels for the source image.
[0086] In formula (2), BCE loss The calculation formula is shown in formula (4):
[0087] (4)
[0088] In formula (4), represents the label of the source image, Represents the encoder pair label The predicted saliency map is obtained by predicting the corresponding source image.
[0089] In this application, in order to further improve the background noise reduction (i.e., background noise reduction) effect of the collaborative salient target detection network based on comprehensive feature mining and background noise reduction, this application constructs a contrastive learning module, and calculates the cosine similarity loss based on the collaborative salient comprehensive features and background comprehensive features output by the contrastive learning module and the comprehensive features output by the comprehensive feature generation module I. , cosine similarity loss Used to constrain the update of parameters of the backbone network, residual feature module and comprehensive feature generation module I in the training process of collaborative salient target detection network based on comprehensive feature mining and background noise reduction, cosine similarity loss In the process of constraining the above-mentioned parameter updates, the effect of suppressing background noise by the collaborative salient target detection network based on comprehensive feature mining and background noise reduction can be improved, so that the background noise can be reduced, and finally the collaborative salient target detection network based on comprehensive feature mining and background noise reduction can obtain a predicted salient map with low background noise.
[0090] In formula (2), the cosine similarity loss The calculation formula is shown in formula (5):
[0091] (5)
[0092] In formula (5), represents the cosine similarity between the comprehensive features output by the comprehensive feature generation module I and the collaboratively significant comprehensive features output by the comprehensive feature generation module II, represents the cosine similarity between the comprehensive features output by the comprehensive feature generation module I and the background comprehensive features output by the comprehensive feature generation module II, is a very small constant, The purpose of setting is to ensure that the input The value of is not zero. The value of is 1×10 −5 .
[0093] In formula (5), and The calculation formula is shown in equation (6) and equation (7):
[0094] (6)
[0095] (7)
[0096] In formula (6), represents the comprehensive features output by comprehensive feature generation module I, represents the collaboratively significant comprehensive features output by the comprehensive feature generation module II;
[0097] In formula (7), represents the comprehensive features output by comprehensive feature generation module I, Represents the background comprehensive features output by comprehensive feature generation module II.
[0098] In formula (2), the generator loss The calculation formula is shown in formula (8):
[0099] (8)
[0100] In formula (8), represents the probability value that the discriminator predicts that image Gm is image Ggt, Representation and A tensor of all 1s of the same size.
[0101] In this application, the total adversarial loss of the collaborative salient object detection network based on comprehensive feature mining and background denoising The calculation formula is shown in formula (9):
[0102] (9)
[0103] In formula (9), is the discriminator loss, is a hyperparameter. Set to 1;
[0104] In formula (9), the discriminator loss The calculation formula is shown in formula (10):
[0105] (10)
[0106] In formula (10), represents the loss of the label corresponding to the source image processed by the discriminator, represents the loss of the discriminator processing the predicted saliency map output by the decoder,
[0107] In formula (10), The calculation formula is shown in formula (11): The calculation formula for denoting is shown in formula (12):
[0108] (11)
[0109] (12)
[0110] In formula (11), represents the probability value that the discriminator predicts that image Ggt is image Ggt, Representation and All-1 tensors of the same size;
[0111] In formula (12), represents the probability value that the discriminator predicts that image Gm is image Ggt, Representation and A tensor of all zeros of the same size.
[0112] S4. Obtain a sub-training set based on the total training set, and generate a loss in the total and the total adversarial loss Under the guidance of , the sub-training set is used to train the collaborative salient target detection network based on comprehensive feature mining and background noise reduction, and the collaborative salient target detection network model based on comprehensive feature mining and background noise reduction is obtained; specifically, the following steps are included:
[0113] S4-1, randomly selecting 8 images and their corresponding labels for each category of collaborative salient targets from the total training set described in step S1, and then scaling the above images and labels to sizes of 256×256×3 and 256×256×1, respectively, to obtain sub-training sets; in the training process of the collaborative salient target detection network based on comprehensive feature mining and background denoising described in the present application, a sub-training set is input in each training epoch, and the sub-training sets input in different training epochs are all obtained by adopting step S4-1;
[0114] Step S4-2: Input the sub-training set described in step S4-1 into the collaborative salient target detection network based on comprehensive feature mining and background noise reduction described in this application, perform forward propagation, and calculate the total generation loss of the collaborative salient target detection network based on comprehensive feature mining and background noise reduction respectively and the total adversarial loss , in total generated losses and the total adversarial loss Back propagation is performed under the guidance of , and the weight parameters and hyper parameters of the collaborative salient target detection network based on comprehensive feature mining and background noise reduction are updated, among which the total generation loss The weight parameters and hyperparameters of the constrained backbone network, residual feature module, comprehensive feature generation module I, comprehensive feature generation module II, feature enhancement module and decoder are updated, and the parameter updates of comprehensive feature generation module I and comprehensive feature generation module II are kept consistent. The total adversarial loss The weight parameters and hyperparameters of the discriminator in the constrained adversarial learning module are updated. After 220 epochs, the training process of the collaborative salient object detection network based on comprehensive feature mining and background noise reduction is completed, and the collaborative salient object detection network model based on comprehensive feature mining and background noise reduction is obtained; among them, the hyperparameters include the learning rate and the number of iterations.
[0115] In this embodiment, the training process of the collaborative salient target detection network based on comprehensive feature mining and background noise reduction is implemented based on the NVIDIA 3090 GPU chip, and the Adam optimizer is used to optimize the loss gradient and back propagate. In the training process of the collaborative salient target detection network based on comprehensive feature mining and background noise reduction, the batch size is set to 8, and the learning rates of the residual feature learning module, the comprehensive feature generation module I, the comprehensive feature generation module II, the feature enhancement module, the decoder, and the adversarial learning module are all set to 1×10 -4 , the learning rate of the backbone network is set to 2.5×10 -6The learning rate mode of the backbone network, residual feature learning module, comprehensive feature generation module I, comprehensive feature generation module II, feature enhancement module, decoder and adversarial learning module adopts exponential learning rate decay, where the decay coefficient gamma is set to 0.996.
[0116] S5. Input the image to be detected into the trained collaborative salient target detection network model based on comprehensive feature mining and background noise reduction, and forward propagate once to obtain the predicted salient map and the predicted probability value. The predicted probability value refers to the probability value of image Gm being image Ggt and the probability value of image Ggt being image Ggt.
[0117] test:
[0118] A test set is constructed, wherein the test set consists of a CoCA dataset and a CoSal2015 dataset; wherein the CoCA dataset is an existing dataset, and the URL for obtaining it is: https: / / zhaozhang.net / coca.html, and the CoCA dataset contains 80 categories of collaborative salient targets and 1295 images with mask information; the CoSal2015 dataset is an existing dataset, and the URL for obtaining it is: http: / / www.escience.cn / people / JunweiHan / Co-saliency.html, and the CoSal2015 dataset contains 50 categories of collaborative salient targets and 2015 images with mask information.
[0119] In order to compare the detection effect of the collaborative salient target detection method based on comprehensive feature mining and background noise reduction described in this application, this application compares the five existing collaborative salient target detection methods with the collaborative salient target detection method described in this application, wherein the five existing collaborative salient target detection methods include: DCFM method (from "Democracy Does Matter: Comprehensive Feature Mining for Co-Salient Object Detection"), CoGEM method (from "Co-Salient Object Detection with Uncertainty-Aware Group Exchange-Masking"), DMT method (from "Discriminative Co-Saliency and Background Mining Transformer for Co-Salient Object Detection"), GCoNet+ method (from "GCoNet+: A Stronger Group Collaborative Co-Salient Object Detector"), UFGS method (from "A Unified Transformer Framework for Group-Based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection"), The above five existing collaborative salient target detection methods and the collaborative salient target detection method based on comprehensive feature mining and background noise reduction described in this application are tested using the above test set to obtain predicted salient maps. Then, the predicted salient maps and their corresponding labels are respectively input into the collaborative salient target detection evaluation tool based on PyTorch shown in https: / / github.com / zzhanghub / eval-co-sod to output the evaluation index 、 、 and , evaluation index 、 、 and The results are shown in Table 1. The collaborative salient target detection method based on comprehensive feature mining and background noise reduction described in this application is represented by Ours method in Table 1.
[0120] Table 1 shows the evaluation indicators obtained by testing the above five existing detection methods and the detection method described in this application using the above test set. 、 、 and Results
[0121]
[0122] The evaluation indicators used in this application include , , and ,in, It is used to measure the mean absolute error between the unbinarized predicted saliency map and the label; It is the weighted harmonic mean used to evaluate precision and recall, and the best value of the entire test set (i.e., the best weighted harmonic mean) is defined as ,in ; The similarity between the predicted saliency map and the label is evaluated from the local and global views. The best value of the entire test set (i.e., the best similarity value among the similarities between the predicted saliency map and the label evaluated from the local and global views) is defined as ; is the structural similarity measure between the predicted saliency map and its corresponding label, where 0.5. Among the above four test indicators, The smaller the value, the better. The remaining three indicators , and The larger the value, the better.
[0123] The most challenging part of this application is to test the CoCA dataset in the test set, because the CoCA dataset has complex background information, more occlusions, more targets, and more interference information; however, from the test results shown in Table 1, the collaborative salient object detection method described in this application performs better in all four evaluation indicators on the CoCA dataset. , , and The test results are good. Specifically, the GCoNet+ method is tested to obtain The lowest value, value, Value and The value is the highest. Therefore, for the test of the CoCA dataset, this application focuses on comparing the test results of the GCoNet+ method with the collaborative salient target detection method described in this application, as follows: Compared with the GCoNet+ method, the collaborative salient target detection method described in this application has The evaluation index is reduced by (0.081-0.074) / 0.081=8.64%; compared with the GCoNet+ method, the collaborative salient object detection method described in this application is The evaluation index is improved by (0.658-0.637) / 0.637=3.30%; compared with the GCoNet+ method, the collaborative salient object detection method described in this application is The evaluation index has not been improved; compared with the GCoNet+ method, the collaborative salient object detection method described in this application The evaluation index increased by (0.753-0.738) / 0.738=2.03%. Obviously, from the above test results, it can be seen that the collaborative salient object detection method described in this application obtains The evaluation index is significantly reduced, indicating that the predicted salient map obtained by the collaborative salient target detection method described in this application when detecting the CoCA dataset is more accurate than the predicted salient map obtained by the above five existing collaborative salient target detection methods. The predicted salient map obtained by the collaborative salient target detection method described in this application has less deviation from the label corresponding to the source image. The collaborative salient target detection method described in this application is more outstanding in capturing the details of the collaborative salient target and can have higher detection accuracy. In addition, the collaborative salient target detection method described in this application is more accurate. The evaluation index is significantly reduced, which also shows that the collaborative salient target detection method described in the present application can still effectively capture the collaborative salient target from the image when detecting a data set with more background noise. This also shows that the collaborative salient target detection method described in the present application has good anti-interference ability when detecting a data set with more background noise; in addition, the collaborative salient target detection method described in the present application The evaluation index is significantly reduced, which also shows that the collaborative salient target detection method described in this application can effectively capture collaborative salient targets of different categories.
[0124] Moreover, when the collaborative salient target detection method described in the present application detects the CoCA dataset, and There is also a good improvement in these two evaluation indicators, which shows that the predicted salient map obtained by the collaborative salient target detection method described in this application has a high accuracy and recall rate, and the structural similarity between the predicted salient map and its corresponding label is high.
[0125] In addition, the collaborative salient object detection method described in this application also has significant improvements on the CoSal2015 dataset in the test set, as follows:
[0126] For the test of CoSal2015 dataset, the DMT method is tested among the above five existing collaborative salient object detection methods. , , and The test results are good. Specifically, the DMT method is used to test the The lowest value, value, Value and The value is the highest. Therefore, for the test of CoSal2015 dataset, this application focuses on comparing the test results of the DMT method with the collaborative salient target detection method described in this application. The details are as follows: Compared with the DMT method, the collaborative salient target detection method described in this application has The evaluation index is reduced by (0.045-0.035) / 0.045=22.22%; compared with the DMT method, the collaborative salient target detection method described in this application is The evaluation index is improved by (0.917-0.905) / 0.905=1.33%; compared with the DMT method, the collaborative salient object detection method described in this application is The evaluation index is improved by (0.944-0.936) / 0.936=0.85%; compared with the DMT method, the collaborative salient object detection method described in this application is The evaluation index is improved by (0.906-0.897) / 0.897=1.00%. The collaborative salient object detection method described in this application can also achieve very good results when tested on the CoSal2015 dataset, especially The evaluation index is significantly reduced, which shows that the predicted salient map obtained by the collaborative salient target detection method described in the present application when detecting the CoSal2015 dataset is more accurate than the predicted salient maps obtained by the above five existing collaborative salient target detection methods. The predicted salient map obtained by the collaborative salient target detection method described in the present application has less deviation from the label corresponding to the source image. The collaborative salient target detection method described in the present application is more prominent in capturing the details of collaborative salient targets and can have higher detection accuracy.
Claims
1. A collaborative salient object detection method based on comprehensive feature mining and background noise reduction, characterized in that: The steps include: S1, obtain the total training set; S2. Constructing a collaborative salient target detection network based on comprehensive feature mining and background noise reduction; the collaborative salient target detection network based on comprehensive feature mining and background noise reduction comprises an input layer, a backbone network, a residual feature module, a comprehensive feature generation module I, a feature enhancement module, a decoder and an adversarial learning module connected in sequence; the backbone network comprises a feature extraction module I, a feature extraction module II, a feature extraction module III and a feature extraction module IV connected in sequence, and the output ends of the feature extraction module I, the feature extraction module II and the feature extraction module III are all connected to the input end of the decoder; the output end of the backbone network is also connected to the contrast learning module, and the output end of the residual feature module is also connected to the feature enhancement module; the output end of the input layer is also connected to the input end of the adversarial learning module; S3. Constructing the total generation loss of a collaborative salient object detection network based on comprehensive feature mining and background denoising and the total adversarial loss ; S4. Obtain a sub-training set based on the total training set, and generate a loss in the total and the total adversarial loss Under the guidance of , the sub-training set is used to train the collaborative salient target detection network based on comprehensive feature mining and background noise reduction, and the collaborative salient target detection network model based on comprehensive feature mining and background noise reduction is obtained; S5. Input the image to be detected into the trained collaborative salient object detection network model based on comprehensive feature mining and background noise reduction, and forward propagate once to obtain the predicted salient map and predicted probability value; In step S2, the feature enhancement module includes three element-by-element multiplication modules, two matrix multiplication modules and two Add layers, wherein the first element-by-element multiplication module is connected to the output end of the element-by-element multiplication module in the comprehensive feature generation module I and the output end of the Add layer in the residual feature module, and the second element-by-element multiplication module is connected to the output end of the Add layer in the residual feature module and the output end of the response module in the comprehensive feature generation module I; the output ends of the first element-by-element multiplication module and the second element-by-element multiplication module are both connected to the first Add layer, and the first Add layer is sequentially connected to the MCA attention module, the convolution layer with a convolution kernel size of 1×1, and the ReLU activation layer, and the ReLU activation layer is respectively connected to The Key convolution layer, Query convolution layer and Value convolution layer are connected. The Key convolution layer and the Query convolution layer are connected to the first matrix multiplication module. The first matrix multiplication module is connected to the descending sorting module and the Softmax layer respectively. The output ends of the descending sorting module and the Softmax layer are connected to the input end of the third element-by-element multiplication module. The output end of the third element-by-element multiplication module and the output end of the Value convolution layer are connected to the input end of the second matrix multiplication module. The output end of the second matrix multiplication module and the output end of the ReLU activation layer are connected to the input end of the second Add layer. The output end of the second Add layer is connected to the input end of the decoder.
2. The collaborative salient object detection method based on comprehensive feature mining and background noise reduction according to claim 1, characterized in that: Step S1 specifically includes the following specific steps: S1-1, obtain the original training set; S1-2, perform data enhancement on the images in the original training set to obtain the total training set, specifically including the following steps: (1), flip the images in the original training set left and right, with the probability of each image being flipped being 50%, to obtain a primary enhanced data set; (2), flip the images in the primary enhanced data set up and down, with the probability of each image being flipped being 50%, to obtain a secondary enhanced data set; (3), perform color enhancement on all images in the secondary enhanced data set to obtain the total training set; wherein the color enhancement includes brightness enhancement, contrast enhancement, color enhancement and sharpness enhancement.
3. The collaborative salient object detection method based on comprehensive feature mining and background noise reduction according to claim 1, characterized in that: In step S2, the residual feature module includes a convolution layer and an Add layer with a convolution kernel size of 1×1 connected in sequence, and the input end of the convolution layer with a convolution kernel size of 1×1 is also connected to the input end of the Add layer; wherein, the convolution layer with a convolution kernel size of 1×1 in the residual feature module is used to perform a convolution operation on the feature map output by the feature extraction module IV; the Add layer is used to perform a pixel-by-pixel addition operation on the feature map output by the feature extraction module IV and the feature map output by the convolution layer to obtain a feature map with residual features.
4. The collaborative salient object detection method based on comprehensive feature mining and background noise reduction according to claim 1, characterized in that: In step S2, the comprehensive feature generation module I includes two parallel convolution layers with a convolution kernel size of 1×1, the input ends of the two convolution layers with a convolution kernel size of 1×1 are connected to the output end of the Add layer of the residual feature module, and the output ends of the two convolution layers with a convolution kernel size of 1×1 are connected to the matrix multiplication module, and the matrix multiplication module is sequentially connected to the seed selection module, the response module, and the element-by-element multiplication module; wherein, the input end of the seed selection module is connected to the output end of the Add layer of the residual feature module, and the input end of the element-by-element multiplication module is connected to the output end of the Add layer of the residual feature module.
5. The collaborative salient object detection method based on comprehensive feature mining and background noise reduction according to claim 1, characterized in that: In step S2, the contrastive learning module includes two element-by-element multiplication modules and a comprehensive feature generation module II connected to the output ends of the two element-by-element multiplication modules, the input ends of the two element-by-element multiplication modules are connected to the output end of the feature extraction module IV in the backbone network, and the input ends of the two element-by-element multiplication modules are also connected to the Input layer respectively; wherein, the structure of the comprehensive feature generation module II is the same as the network structure of the comprehensive feature generation module I, and the function is also the same.
6. The collaborative salient object detection method based on comprehensive feature mining and background noise reduction according to claim 1, characterized in that: In step S2, the adversarial learning module includes an input layer, two element-by-element multiplication modules, and a discriminator. The input layer of the adversarial learning module is connected to one of the element-by-element multiplication modules, and the other element-by-element multiplication module is connected to the output end of the decoder. The two element-by-element multiplication modules are also connected to the output end of the input layer connected to the backbone network, and the output ends of the two element-by-element multiplication modules are connected to the discriminator; the discriminator includes four discriminant modules connected in sequence, a fully connected layer, and a Sigmoid layer, wherein the four discriminant modules have the same structure and function.
7. The collaborative salient object detection method based on comprehensive feature mining and background noise reduction according to claim 1, characterized in that: In step S3, the total generation loss of the collaborative salient object detection network based on comprehensive feature mining and background noise reduction is Including IoU loss 、BCE loss , cosine similarity loss and the generator loss Total generation loss It is used to constrain the update of weight parameters and hyperparameters of the backbone network, residual feature module, comprehensive feature generation module I, comprehensive feature generation module II, feature enhancement module and decoder, and the parameter updates of comprehensive feature generation module I and comprehensive feature generation module II are consistent; total adversarial loss Used to constrain the update of the weight parameters and hyperparameters of the discriminator in the adversarial learning module.
8. The collaborative salient object detection method based on comprehensive feature mining and background noise reduction according to claim 1, characterized in that: Step S4 specifically includes the following steps: S4-1, randomly selecting N images and their corresponding labels for each category of collaborative salient objects from the total training set described in step S1, where N is a natural number, and then scaling the images and labels to sizes of 256×256×3 and 256×256×1, respectively, to obtain a sub-training set; Step S4-2: Input the sub-training set described in step S4-1 into the collaborative salient object detection network based on comprehensive feature mining and background noise reduction, perform forward propagation, and calculate the total generation loss of the collaborative salient object detection network based on comprehensive feature mining and background noise reduction respectively. and the total adversarial loss , in total generated losses and the total adversarial loss Back propagation is performed under the guidance of , and the weight parameters and hyper parameters of the collaborative salient target detection network based on comprehensive feature mining and background noise reduction are updated, among which the total generation loss The weight parameters and hyperparameters of the constrained backbone network, residual feature module, comprehensive feature generation module I, comprehensive feature generation module II, feature enhancement module and decoder are updated, and the parameter updates of comprehensive feature generation module I and comprehensive feature generation module II are kept consistent. The total adversarial loss The weight parameters and hyperparameters of the discriminator in the constrained adversarial learning module are updated. After 220 epochs, the training process of the collaborative salient object detection network based on comprehensive feature mining and background noise reduction is completed, and the collaborative salient object detection network model based on comprehensive feature mining and background noise reduction is obtained; among them, the hyperparameters include the learning rate and the number of iterations.
9. The collaborative salient object detection method based on comprehensive feature mining and background noise reduction according to claim 8, characterized in that: In step S4-1, during the training process of the collaborative salient object detection network based on comprehensive feature mining and background noise reduction, a sub-training set is input in each training epoch, and the sub-training sets input in different training epochs are all obtained by using step S4-1.
Citation Information
Patent Citations
Underwater acoustic target recognition method based on adversarial residual network
CN113435276A
Cooperative saliency detection method guided by weak supervised learning
CN114743027A
Image fusion method based on multi-scale feature extraction and target reconstruction
CN117876836A
Method for reconstructing abnormal behavior detection model of generative adversarial network
CN117876959A
Audio source feature separation and target audio source generation
US20230352040A1
