A weakly supervised saliency detection method and training strategy based on mixed labels

Abstract

The present application relates to a kind of weakly supervised saliency detection method and training strategy based on mixed label.The present application proposes a two-stage network to respectively correct rough label and the detection framework and method of saliency object in RGB image: in the correction network, mixed module with guide and aggregation mechanism is designed, and feature is aggregated and corrected in different stages;In addition, a special iterative training strategy is proposed, which realizes the full use of accurate label.The test framework and method described in the present application have achieved competitive performance on multiple public benchmark datasets.Under the condition of multi-target, complex background, low contrast and the like, the present application can have better prediction results under the support of rough label.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a weakly supervised saliency detection method based on mixed labels and a training strategy. BACKGROUND

[0002] As the most direct information that people can intuitively perceive, images can contain more information than human-processed text. Therefore, people often use images for important activities such as information acquisition, information expression, and information transmission. The images that the human eye pays attention to from an image often have a sequence, and the images that the human eye pays attention to first can best reflect the part that the human eye pays most attention to. Saliency detection is such an image processing technology. Detecting the most eye-catching objects from an RGB color photo or a video helps to quickly extract the information of an image, which can greatly help image retrieval and improve the efficiency of image retrieval in the Internet era with a large amount of information. In addition, information transmission in the Internet era is also complex. How to transmit more information under the same bandwidth often represents a more real network experience. Through saliency detection technology, the main objects can be extracted from the image or video, and the image or video can be compressed, thereby reducing the size of the image or video and improving the network transmission efficiency. These applications are only a part of saliency object detection. For example, saliency object detection technology is also useful in the fields of target tracking, image / video editing, etc. Early scholars generally used mathematical and statistical methods for saliency object detection. With the rapid development of deep learning technology, image processing technologies such as saliency object detection have undergone a qualitative leap. With the help of deep learning technology, the detection accuracy has undergone a qualitative leap. In the field of target detection, the computer has for the first time surpassed humans and has higher accuracy. However, the problem that follows is that deep learning often needs to provide accurate labeling information for a type of object for neural network learning. These accurate labels are very time-consuming, not only requiring professional software, but also requiring a large amount of manpower. On this basis, weakly supervised methods have emerged. Weakly supervised saliency object detection aims to use simpler labeling information (such as picture-level labeling, scribble labeling, and rough labeling). These labeling information requires a relatively low labeling cost and can usually be completed in a few seconds. In this way, large amounts of data sets can be labeled.

[0003] However, the weak supervision method used in the existing method is usually supervised using sparse labels. Sparse labels such as scribble labels only provide partial accurate information, and a small amount of labels need to be expanded, which will inevitably cause errors and introduce a large amount of noise. In addition, the existing method usually adopts a single-stage training strategy, which is difficult to optimize the selection of labels. SUMMARY

[0004] In view of the defects in the prior art, the purpose of the present application is to provide a weakly supervised saliency detection method based on mixed labels. Specifically, the present application uses a large number of coarse labels and a small number of real labels as supervision, and decouples the task into two sub-tasks of coarse label refinement and salient object detection, and then designs the corresponding correction network (R-Net) and saliency prediction network (S-Net). Among them, R-Net designs a mixer module with guidance and aggregation mechanism to realize two-stage feature decoding. The guidance stage is used to introduce guidance information (such as the location information and integrity of the target) from the RGB image guidance branch to ensure the robustness of the baseline, and the aggregation stage dynamically integrates features of different levels according to its correction or complementary effect.

[0005] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0006] A weakly supervised saliency detection method based on mixed labels, characterized by R-Net and S-Net.

[0007] R-Net adopts an encoder-decoder architecture as a whole to receive information from the main branch and the guidance branch, forming a double-flow encoding structure. The R-Net also includes a mixer BGA with guidance and aggregation mechanism to realize feature decoding in the guidance stage and the aggregation stage.

[0008] S-Net is used for saliency detection of the RGB picture to be predicted under the supervision of real labels.

[0009] The above R-Net refers to the correction network, and the S-Net refers to the saliency prediction network. The main branch refers to the main repair branch, including the RGB picture and the coarse label. The guidance branch refers to an independent RGB picture guidance branch.

[0010] The above guidance stage refers to using the information of the guidance branch to supplement the main branch. The aggregation stage refers to integrating the encoder features of the corresponding layer, the decoder features of the previous layer and the global features from the top layer of the encoder.

[0011] On the basis of the above scheme,

[0012] In the guidance stage, the encoders of the corresponding layers in the main branch and the guidance branch are connected in series for supplementation, and then the necessary channel features are highlighted for filtering using channel attention, which is expressed as formula (1):

[0013]

[0014] In the above formula, represents the complementary features after channel attention processing, CA is the channel attention operation, and respectively represent the encoder feature of the i-th layer in the main branch and the guidance branch, and represents the concatenation operation along the channel dimension. represents the element-wise multiplication with channel dimension broadcasting;

[0015] The spatial position mask that needs to be emphasized is generated from the perspective of RGB information, and is used to update the feature of the refinement branch, which is expressed as formula (2):

[0016]

[0017] In the above formula, is the encoder feature of the final output of the guidance stage after spatial attention enhancement, SA is the spatial attention operation, is the element-wise multiplication operation, Conv 1×1 represents a convolution layer with a kernel size of 1x1.

[0018] On the basis of the above scheme,

[0019] In the aggregation stage, the encoder feature of the corresponding layer generated by the guidance stage, the global feature from the top layer encoder and the decoder feature of the previous layer are integrated, which is specifically:

[0020] The semantic features of the main branch and the guidance branch and the encoder features generated by the guidance stage are combined through the importance weighting strategy, which is expressed as formula (3):

[0021]

[0022] In the above formula, represents the fused semantic feature from the two branches, that is, the global feature from the top layer encoder; is the encoder feature of the corresponding layer; P i is the importance weight of learning, which is used to control f g and the fusion proportion of the feature;

[0023] Then, the fused feature containing global semantic information is activated as a semantic mask, which is used to correct the up-sampled decoder feature, which is expressed as formula (4):

[0024]

[0025] In the above formula, is the corrected decoder feature of the i-th layer; represents the original decoder feature of the i+1-th layer, that is, the decoder feature of the previous layer; Up represents the up-sampling operation of bilinear interpolation, and sigma represents the sigmoid activation function;

[0026] The filtered features are further supplemented to the corrected decoder features through a spatial attention mechanism to obtain more comprehensive saliency-related decoder features, expressed as formula (5):

[0027]

[0028] In the above formula, is the i-th layer corrected decoder feature, and SA represents a spatial attention operation.

[0029] Another object of the present application is to provide a weakly supervised saliency detection training strategy based on mixed labels.

[0030] To achieve the above object, the technical solution adopted by the present application is:

[0031] A weakly supervised saliency training strategy based on mixed labels, characterized by comprising the following steps:

[0032] Step 1, randomly select a certain number of samples in the data set as the training subset of the true label, and use the minimum grid method to generate the corresponding rough label of all samples in the above data set;

[0033] Step 2, divide all training samples including true labels and rough labels obtained in step 1 into n groups by averaging, wherein the true labels are all assigned to group 1, and the other rough labels are assigned to groups 2 to n;

[0034] Step 3, use the training samples of each group obtained in step 2 to train R-Net and S-Net using an alternating incremental iteration mechanism until all training samples are traversed;

[0035] Step 4, directly input the RGB picture into the S-Net network, and the S-Net can output the saliency object detection result according to the input picture.

[0036] On the basis of the above scheme, the process of training R-Net and S-Net using an alternating incremental iteration mechanism in step 3 is as follows:

[0037] Step 3-1, use the samples with true labels in group 1 to train R-Net, use the trained R-Net to test the samples in group 2 to obtain the corresponding pseudo labels, and send the samples with true labels in group 1 and the samples with pseudo labels in group 2 into S-Net for retraining S-Net; use the retrained S-Net to test the samples in group 3 to obtain pseudo labels for the next iteration of R-Net training;

[0038] Step 3-2, the samples with pseudo labels in group k are continuously sent into the R-Net for training, and the R-Net after retraining is used to test the samples in group k+1 to obtain corresponding pseudo labels, and the samples with pseudo labels in group k+1 are continuously sent into the S-Net for retraining; the S-Net after retraining is used to test the samples in group k+2 to obtain pseudo labels required for R-Net training in the next iteration;

[0039] Step 3-3, the R-Net and the S-Net are alternately iteratively trained according to the steps described in step 3-2 until the training is completed;

[0040] The above-mentioned training completion flag is that all training samples are traversed, that is, all training samples are sent into the R-Net and the S-Net for training;

[0041] In the above-mentioned step 3-2, the initial value of k is 3; according to step 3-3, the value of k in step 3-2 is increased by 2 after each iteration.

[0042] On the basis of the above-mentioned scheme,

[0043] After each iteration, the mean absolute error MAE is used to implement a credibility verification mechanism for the R-Net and the S-Net, and the credibility verification mechanism is specifically:

[0044] The mean absolute error MAE scores of the R-Net after the tth iteration and the R-Net after the (t-1) th iteration are calculated on the validation set and compared, and the R-Net with a smaller mean absolute error MAE score is selected to test the samples in group 2t and generate corresponding pseudo labels;

[0045] The mean absolute error MAE scores of the S-Net after the tth iteration and the S-Net after the (t-1) th iteration are calculated on the validation set and compared, and the S-Net with a smaller mean absolute error MAE score is selected to test the samples in group 2t+1 and generate corresponding pseudo labels;

[0046] The initial value of t is 2; and 2t+1<n, n refers to the number of training sample groups in step 2.

[0047] The weakly supervised saliency detection method and the training strategy based on mixed labels have the following beneficial effects:

[0048] 1.The present application explores a new weakly supervised salient object detection task based on mixed labels, and proposes a two-stage network to correct the coarse labels and predict the salient objects in RGB images respectively. To this end, in the correction network, the present application designs a mixer module with guidance and aggregation mechanism to aggregate and correct the features at different stages. In addition, the present application proposes a special iterative training strategy for this new task to make full use of the accurate labels.

[0049] 2.The method of the present application achieves competitive performance on multiple public benchmark datasets. Under conditions such as multi-target, complex background and low contrast, the present application can have better prediction results under the support of coarse labels. BRIEF DESCRIPTION OF DRAWINGS

[0050] The present application has the following drawings:

[0051] Figure 1 The network overall architecture diagram of the weakly supervised salient object detection method based on mixed labels of the present application;

[0052] Figure 2 The visualization result diagram of the weakly supervised salient object detection method based on mixed labels of the present application. DETAILED DESCRIPTION

[0053] The present application will be further described in detail below in combination with the drawings.

[0054] The network overall architecture of the weakly supervised salient object detection method based on mixed labels is as shown in Figure 1 The two networks are composed of a correction network (R-Net) and a salient prediction network (S-Net). The two networks cooperate with each other for alternating training. During training, the two networks use an alternating incremental iteration mechanism to solve the imbalance problem between real label data and pseudo label data, and use a reliability verification mechanism to ensure that the two networks can provide reliable labels to each other.

[0055] The prediction network in the two sub-networks designed by the present application is replaceable, so the present application focuses on the design of the correction network. The correction network is composed of two stages of guidance and aggregation, and the overall encoder-decoder architecture is adopted, and the encoder adopts a double-flow structure. Specifically, taking the RGB picture and the coarse label as the input, considering the uncertainty and noise of the rough label, an independent RGB picture guidance branch is introduced in the R-Net to form a double-flow encoding structure, which is used to provide some guidance information such as object positioning and integrity to the main branch, so as to ensure that a relatively robust performance baseline can be obtained. In addition, in order to ensure the effect and efficiency of network training, the present application proposes a corresponding training strategy from the aspects of quantity allocation, training method and reliability judgment, and designs an alternating incremental iteration mechanism and a reliability verification mechanism respectively.

[0056] The encoders of the two data streams of the revision network are both based on ResNet-50 to extract corresponding multi-level features. Then, the present application proposes a mixer with guidance and aggregation mechanism (BGA) to realize two-stage feature decoding. The role of the first stage is guidance, that is, to supplement the main stream branch with the information of the guidance branch to ensure a relatively stable baseline performance of the main stream branch. The role of the second stage is aggregation, that is, to integrate the encoder features of the corresponding layer, the decoder features of the previous layer and the global features from the top layer of the encoder by considering the role of different features.

[0057] In the first stage of the revision network, that is, the guidance stage, the present application expects that the RGB branch can provide guidance information (such as object positioning and integrity) for the main stream branch, so as to ensure its effective learning and stable performance baseline.

[0058] Firstly, in order to ensure that sufficient saliency information can be transferred to the main stream branch and to alleviate the unreliable noise from the rough label input, the present application supplements and filters the channel dimension features. Specifically, first, the encoder features of the corresponding layer in the two branches are concatenated for supplementation, and then the channel attention is used to highlight the necessary channel features for filtering. This process can be expressed as:

[0059]

[0060] wherein represents the complementary features after channel attention enhancement, CA is the channel attention operation, and respectively represent the encoder features of the i-th layer in the main stream branch and the guidance branch, and represents the concatenation operation along the channel dimension. represents the element-wise multiplication with channel dimension broadcast.

[0061] Secondly, in addition to the direct supplementation of the channel dimension, the RGB branch can also provide spatial guidance information at the pixel level, which can both strengthen important areas and suppress irrelevant noise interference. Specifically, the present application uses spatial attention to generate a spatial position mask that needs to be emphasized from the perspective of RGB information, and uses it to update the features of the refinement branch:

[0062]

[0063] wherein represents the i-th layer of the encoder features of the final output of the guidance stage after spatial attention enhancement, SA is the spatial attention operation, is the element-wise multiplication operation, and Conv 1×1 represents a convolution layer with a convolution kernel size of 1×1.

[0064] The second stage is mainly used to realize the aggregation of multi-level features, integrating the encoder features of the corresponding layer generated in the first stage, the global features from the top-level encoder, and the decoder features of the previous layer. In order to more effectively realize the aggregation, it is necessary to analyze the role of various features. Generally speaking, in the feature decoding stage, both the encoder features and the global features should play a supporting role to obtain better decoder features. The supporting role can be divided into two aspects: one is to perfect the decoder features under the guidance of global information; the second is to supplement the decoder features under the guidance of the encoder features.

[0065] Firstly, the global features from the top-level encoder layer are crucial for distinguishing prominent objects, but as the decoding process proceeds, the semantic constraints will gradually be attenuated. Therefore, in order to enforce semantic information throughout the decoding process, the present application generates a corresponding semantic guidance mask to refine the decoder features of each layer. Specifically, the present application first combines the semantic features from the two branches and the encoder features generated in the first stage through an importance weighting strategy:

[0066]

[0067] wherein represents the fused semantic features from the two branches, that is, the global features from the top-level encoder; is the encoder feature of the corresponding layer; P i is the importance weight learned to control the g and fusion proportion of the features. Then, the fused features containing global semantic information are activated as a semantic mask to correct the upsampled decoder features:

[0068]

[0069] wherein is the corrected decoder feature of the i-th layer; represents the original decoder feature of the i+1-th layer, that is, the decoder feature of the previous layer; Up represents the bilinear interpolation up-sampling operation, and σ represents the sigmoid activation function.

[0070] Secondly, since the encoder features contain a lot of valuable information, they can supplement the learning of the decoder features, such as the richer spatial information of the shallower features, which can better restore details, etc. Therefore, the present application further uses the filtered features to supplement the corrected decoder features through a spatial attention mechanism to obtain more comprehensive saliency-related decoder features. This process can be expressed as:

[0071]

[0072] wherein is the i-th layer modified decoder feature, and SA denotes the spatial attention operation.

[0073] The training set (such as the DUTS-TR dataset) used in the present application should contain pixel-level true labels and coarse labels, wherein the coarse labels are only used as inputs of the S-Net and are not supervised. At the same time, during the network training process, the pseudo labels will be generated by the network as supervision information. In the implementation process, the present application randomly selects 1000 samples from the DUTS-TR dataset containing 10000 samples as a training subset with true labels, and uses the minimum barrier salient object detection method (J. Zhang, S. Sclaroff, Z. Lin, X. Shen, B. Price, and R. Mech, "Minimum barrier salient object detection at 80fps," in Proc. CVPR, 2015, pp. 1404-1412) to generate the corresponding coarse labels of all samples (including the aforementioned 1000 samples) in the DUTS-TR dataset. For such mixed data, in order to ensure the effectiveness and efficiency of network training, the present application proposes two key training mechanisms, including an alternating incremental iteration mechanism and a credibility verification mechanism.

[0074] Alternating incremental iteration mechanism. Specifically, the S-Net of the current iteration is trained using the pseudo labels generated by the R-Net of the current iteration, and the pseudo labels generated by the trained S-Net are further used for the R-Net training of the next iteration. Secondly, in the weakly supervised salient object detection framework with mixed labels, another important problem is the sample imbalance caused by the difference in the number of true label samples and coarse label samples. If the unbalanced training samples are directly used for network training, it may lead to network collapse. Therefore, the two networks are trained in an alternating manner, and the training samples are gradually added to the two networks in an incremental manner until all the training samples are traversed, which is called the alternating incremental iteration mechanism.

[0075] The process of alternating incremental iteration is as follows:

[0076] First, as described above, 1000 samples are randomly selected from the training set (i.e. the dataset, assuming that the total number of samples is 10000) as a training subset with true labels, and the corresponding coarse labels of all samples are generated using the minimum barrier salient object detection method.

[0077] All training samples including true labels and coarse labels are evenly grouped. For example, 10000 samples can be evenly divided into 10 groups, named group 1 to group 10, and each group contains 1000 samples. The 1000 samples in group 1 are samples with true labels.

[0078] The R-Net is trained with the samples with real labels in Group 1, the samples in Group 2 are tested with the trained R-Net to obtain corresponding pseudo labels, and the samples with real labels in Group 1 and the samples with pseudo labels in Group 2 are sent into the S-Net for retraining; the S-Net after retraining is used to test the samples in Group 3 to obtain pseudo labels for the R-Net training in the next iteration;

[0079] In turn, the samples with pseudo labels in Group k are continuously sent into the R-Net for training, the R-Net after retraining is used to test the samples in Group k+1 to obtain corresponding pseudo labels, and the samples with pseudo labels in Group k+1 are continuously sent into the S-Net for retraining; the S-Net after retraining is used to test the samples in Group k+2 to obtain pseudo labels required for the R-Net training in the next iteration;

[0080] The initial value of k is 3; each time the iteration is performed, the value of k is increased by 2 than the last iteration, specifically as follows:

[0081] When k is the initial value 3, it corresponds to the second iteration, that is, the samples with pseudo labels in Group 3 are continuously sent into the R-Net for training, the R-Net after training is used to test the samples in Group 4 to obtain corresponding pseudo labels, and the samples with pseudo labels in Group 4 are continuously sent into the S-Net for retraining; the S-Net after retraining is used to test the samples in Group 5 to obtain pseudo labels required for the R-Net training in the next iteration;

[0082] When the third iteration is performed, the value of k is increased by 2 than the second iteration, that is, 5, and the third iteration is specifically as follows:

[0083] The samples with pseudo labels in Group 5 are continuously sent into the R-Net for training, the R-Net after training is used to test the samples in Group 6 to obtain corresponding pseudo labels, and the samples with pseudo labels in Group 6 are continuously sent into the S-Net for retraining; the S-Net after retraining is used to test the samples in Group 7 to obtain pseudo labels required for the R-Net training in the next iteration;

[0084] The fourth iteration and the iterations thereafter are as described above by analogy until the training is completed, and in this example, the training is completed after 5 iterations.

[0085] The sign of completion of training is that all training samples are added into the network for training, that is, all samples in Groups 1 to 10 are trained in the R-Net or the S-Net.

[0086] In addition, since both the R-Net and the S-Net are trained with mixed labels, in order to ensure the effectiveness and executability of the training, the network is first trained under pseudo-label samples, and then fine-tuned on real-label samples in a training cycle.

[0087] It should be noted that after introducing the alternating incremental iteration mechanism, the number of sample groups needs to be appropriately selected for different sample quantities in the training set.

[0088] The credibility verification mechanism. The purpose of the alternating training of the two networks is to provide better pseudo-labels for each other, so the credibility verification mechanism is introduced from the second iteration, and a verification set including 100 images is tested to ensure the effectiveness of the provided labels. Only when the performance of the current model on the verification set is better than that of the previous best model, the current model is used to generate pseudo-labels for the corresponding group and participate in the next training, specifically:

[0089] The average absolute error MAE scores of the R-Net trained after the tth iteration and the R-Net trained after the (t-1) th iteration on the verification set are compared, and the R-Net with the smaller average absolute error MAE score is selected to test the samples in the test group 2t and generate corresponding pseudo-labels;

[0090] The average absolute error MAE scores of the S-Net trained after the tth iteration and the S-Net trained after the (t-1) th iteration on the verification set are compared, and the S-Net with the smaller average absolute error MAE score is selected to test the samples in the test group 2t+1 and generate corresponding pseudo-labels;

[0091] The initial value of t is 2; and 2t+1 < n, MAE refers to the average absolute error, which represents the error between the prediction and the ground truth, and the smaller the value is, the better it is.

[0092] The detection process of the present application is: the RGB picture is directly input into the trained saliency prediction network S-Net, and the S-Net will automatically detect the saliency object from the picture according to the learned knowledge and output in the form of a mask.

[0093] After inputting the RGB picture into the S-Net, the S-Net aggregation interaction module effectively utilizes the adjacent features through mutual learning, and uses the self-interaction module to adaptively extract multi-scale information from the picture to meet the saliency targets of different physical sizes, and then the S-Net outputs the saliency map. For details of the detection process, refer to the prior art document: L. Wang, H. Lu, Y. Wang, M. Feng, D. Wang, B. Yin, and X. Ruan, “Learning to detect salient objects with image-level supervision,” in Proc. CVPR, 2017, pp. 136-145.

[0094] Figure 2 Visual examples of the present technology are given. The first column is a color image, the second column is the true value map of the RGB saliency target detection, the third column is the rough saliency map generated by the traditional method, and the fourth column is the saliency map predicted by the present application. From the results, it can be seen that the present application method achieves better visual effect in many challenging scenes.

[0095] The contents not described in detail in the specification belong to the prior art known to those skilled in the art.

Claims

1. A method for weakly supervised saliency detection based on mixed labels, characterized in that, The method comprises the following steps: Step 1, randomly selecting a certain number of samples in the data set as a training subset of real labels, and using the minimum grid method to generate the corresponding rough labels of all samples in the above data set; Step 2, divide all the training samples including the real labels and the coarse labels obtained in step 1 into n groups averagely, wherein the real labels are classified into group 1 and the other coarse labels are classified into groups 2 to n ; Step 3, using the training samples obtained in step 2 to train R-Net and S-Net by using an alternating incremental iteration mechanism until all training samples are traversed; Step 4, inputting the RGB picture directly into the S-Net network, and the S-Net can output the saliency detection result according to the input picture; The process of training R-Net and S-Net by using the alternating incremental iteration mechanism in step 3 is specifically: Step 3-1, training R-Net by using the samples with real labels in group 1, obtaining the corresponding pseudo labels by testing the samples in group 2 by using the trained R-Net, and inputting the samples with real labels in group 1 and the samples with pseudo labels in group 2 into S-Net for retraining S-Net; using the retrained S-Net to test the samples in group 3 to obtain the pseudo labels for the next iteration of R-Net training; Step 3-2, the group with pseudo-labels continues to be sent to R-Net for training, and the group samples in are tested by using the retrained R-Net to obtain corresponding pseudo-labels, and the group with pseudo-labels continues to be sent to S-Net for retraining; and the group samples in are tested by using the retrained S-Net to obtain pseudo-labels required for R-Net training in the next iteration; Step 3-3, alternately and iteratively training R-Net and S-Net according to the steps in step 3-2 until the training is completed; The above training completion sign is that all training samples are traversed, that is, all training samples are input into R-Net and S-Net for completion of training; In step 3-2 above, the The initial value is 3; in step 3-3, the value described in step 3-2 is increased by 2 for each iteration The value is increased by 2 from the previous iteration. The above R-Net adopts an encoder-decoder architecture as a whole to receive the information of the main stream branch and the guide branch to form a double-flow coding structure; the R-Net further comprises a mixer BGA with a guide and aggregation mechanism to realize feature decoding in the guide stage and the aggregation stage; S-Net is used for saliency detection of the RGB picture to be predicted under the supervision of real labels; The above R-Net refers to a correction network, S-Net refers to a saliency prediction network; the main stream branch refers to a main stream repair branch, including an RGB picture and a rough label; the guide branch refers to an independent RGB picture guide branch; The above guide stage refers to supplementing the main stream branch by using the information of the guide branch; The aggregation stage refers to integrating the encoder features of the corresponding layer, the decoder features of the previous layer and the global features from the top layer of the encoder.

2. The weakly supervised saliency detection method based on mixed labels according to claim 1, wherein: After each iteration, the R-Net and the S-Net are subjected to a credibility verification mechanism by using a mean absolute error (MAE), which is specifically: The first After the second iteration, the trained R-Net and the... After the second iteration, the trained R-Net calculates and compares the mean absolute error (MAE) scores on the validation set. The R-Net with the smaller MAE score is selected for the test group. The samples in the dataset are used to generate corresponding pseudo-labels; The first After the second iteration, the trained S-Net and the... After the second iteration, the trained S-Net calculates and compares the mean absolute error (MAE) scores on the validation set. The S-Net with the smaller MAE score is selected for the test group. The samples in the dataset are used to generate corresponding pseudo-labels; The above The initial value is 2; and Refers to the number of training sample groups described in step 2.

3. The weakly supervised saliency detection method based on mixed labels according to claim 1, wherein: In the guide stage, the encoders of the corresponding layers in the main stream branch and the guide branch are connected in series for supplementation, and then channel attention is used to highlight the necessary channel features for filtering, which is expressed as formula (1): ； In the above formulae, denotes the complementary feature after channel attention processing, is a channel attention operation, and denote the first layer encoder feature in the main branch and the guidance branch, respectively, denotes a concatenation operation along the channel dimension; denotes an element-wise multiplication with channel dimension broadcasting. A spatial position mask that needs to be emphasized is generated from the RGB information, and the features of the refinement branch are updated based on the mask, which is expressed as formula (2): ； In the above formula, an encoder feature for the final output of the spatial attention enhanced post-guidance stage, a spatial attention operation, an element-wise multiplication operation, denotes a convolutional layer with a kernel size of .

4. The weakly supervised saliency detection method based on mixed labels according to claim 1, wherein: In the aggregation stage, the encoder features of the corresponding layer generated by the integration guide stage, the global features from the top layer encoder and the decoder features of the previous layer are generated, specifically: The semantic features of the mainstream branch and the guide branch and the encoder features generated by the guide stage are combined through an importance weighting strategy, expressed as formula (3): ； In the above formula, denotes the fusion semantic features from two branches, that is, the global features from the top-level encoder; is the encoder feature of the corresponding layer; is the importance weight of learning, used to control and the fusion proportion of features; Then, the fusion features containing global semantic information It is activated as a semantic mask to correct the upsampled decoder features, expressed as Equation (4): ； In the above formula, is the first layer's modified decoder feature; represents the first layer's original decoder feature, i.e., the decoder feature of the previous layer; represents an up-sampling operation of bilinear interpolation, represents a sigmoid activation function; The filtered features are further used to supplement and correct the decoder features to obtain more comprehensive saliency-related decoder features through a spatial attention mechanism, expressed as formula (5): ； In the above formula, is the first layer modified decoder feature, denotes the spatial attention operation.

Images