Cooperative saliency target detection method, system and device based on multi-target constraint and medium

Abstract

The invention discloses a collaborative saliency target detection method, system and device based on multi-target constraint and a medium, and belongs to the field of computer vision, and the method comprises the steps: inputting an image group into an encoder to extract features, and obtaining a feature map; parallel branch learning is carried out on the feature map, a main branch is responsible for learning the overall content of an object in the image to obtain an overall prediction map, and an auxiliary branch is responsible for learning the boundary content of the object in the image to obtain a boundary prediction map; performing adversarial integrity learning on the overall prediction map, and guiding the main branch to adjust the overall prediction map; fusing the boundary prediction map with the adjusted overall prediction map to obtain a fused prediction map; and performing knowledge distillation on the fused prediction map to obtain a collaborative significance prediction map. According to the method, multiple constraints are fully utilized to guide network learning, and the accuracy of collaborative saliency target detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a collaborative salient target detection method, system, device and medium based on multi-target constraints. Background Art

[0002] Co-salient object detection (CoSOD) aims to detect common salient objects in a set of related images. Compared with the standard salient object detection (SOD) task, CoSOD is more challenging and needs to distinguish common objects appearing in different images while identifying distractors.

[0003] Early CoSOD methods often explored the correlation of related images in a group based on handcrafted features. These methods built correlation models by segmenting each image using computational fragments (e.g., superpixels) and mined common regions through sorting mechanisms such as clustering guidance or transformation alignment. At the same time, metric learning, histograms, and contrast statistics were also used to construct better semantic properties for subsequent calculations.

[0004] In the era of deep learning, many end-to-end deep CoSOD models have been proposed. Most of them can be summarized as a four-stage paradigm, which includes (1) extracting image features through an encoder, (2) extracting consensus on a set of image features, (3) dispersing consensus on individual image features, and (4) generating co-saliency masks through a decoder. In addition to the above main branches, recent CoSOD methods tend to introduce auxiliary tasks or side pathways to improve performance, such as additional classification tasks, salient object detection (SOD) priors, various contrastive learning, discriminators, etc. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a collaborative salient target detection method, system, device and medium based on multi-objective constraints. The existing method is improved by only learning the overall consensus features of the object and ignoring the importance of the consensus features of the edge of the object. At the same time, through the way of teachers guiding students in knowledge distillation, the knowledge of teachers from different aspects is integrated, and the idea of ​​multi-objective constraints is introduced.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A collaborative salient object detection method based on multi-objective constraints comprises the following steps:

[0008] Input the image group into the encoder to extract features and obtain a feature map;

[0009] The feature map is learned in parallel. The main branch is responsible for learning the overall content of the object in the image to obtain the overall prediction map, and the secondary branch is responsible for learning the boundary content of the object in the image to obtain the boundary prediction map.

[0010] Conduct adversarial integrity learning on the overall prediction graph to guide the main branch to adjust the overall prediction graph;

[0011] The boundary prediction map is fused with the adjusted overall prediction map to obtain a fused prediction map;

[0012] Knowledge distillation is performed on the fused prediction map to obtain the co-saliency prediction map.

[0013] To optimize the above technical solutions, the specific measures taken also include:

[0014] Furthermore, the image group consists of N different types of images {G1,…,G i ,…,G N} is stacked in the batch dimension, G i represents the i-th category image;

[0015] The encoder is PyramidVisionTransformerv2. The image group G is input into the pre-trained PyramidVision Transformerv2 for feature extraction. Pyramid VisionTransformer v2 has 4 network stages, and each network stage generates feature maps of different scales. represents the feature map output by the first network stage of the encoder, represents the feature map output by the second network stage of the encoder, represents the feature map output by the third network stage of the encoder, Feature map representing the output of the fourth network stage of the encoder.

[0016] Furthermore, the parallel branch learning is specifically as follows:

[0017] The main branch includes a main branch convolution layer, a main branch group consensus aggregation module, a main branch memory comparison module and a main branch decoder;

[0018] The side branch includes a side branch convolution layer, a side branch group consensus aggregation module, a side branch memory comparison module and a side branch decoder;

[0019] The main branch decoder has 4 network stages, and the feature map output by the first network stage of the encoder is Feature map output from the second network stage of the encoder Feature map output by the third network stage of the encoder The feature graph output by the main branch convolution layer is convolved and added with the network stage corresponding to the main branch decoder;

[0020] The sub-branch decoder has 4 network stages, and the feature map output by the first network stage of the encoder is Feature map output from the second network stage of the encoder Feature map output by the third network stage of the encoder The feature graph output by the sub-branch convolution layer is convolved and added with the network stage corresponding to the sub-branch decoder;

[0021] Feature map output by the fourth network stage of the encoder The consensus features are extracted by the main branch group consensus aggregation module, and then updated by the main branch memory comparison module and input into the main branch decoder to generate the overall prediction graph;

[0022] Feature map output by the fourth network stage of the encoder The consensus features are extracted by the sub-branch group consensus aggregation module, and then updated by the sub-branch memory comparison module and input into the sub-branch decoder to generate a boundary prediction map.

[0023] Furthermore, the specific process of extracting consensus features by the main branch group consensus aggregation module and the specific process of extracting consensus features by the secondary branch group consensus aggregation module are:

[0024] The feature map output by the fourth network stage of the encoder In the batch dimension, we get Represents the feature map of class i image, The attention calculation is performed in the non-local block of the input group consensus aggregation module, and the output of the non-local block is recorded as Represents the consensus feature of class i image, which is the same as the input After corresponding multiplication, the consensus features output by the final group consensus aggregation module are obtained by re-joining them in the batch dimension. The specific expression is:

[0025]

[0026] In the formula, concatenate(·) represents the concatenation operation, and n is the batch size; represents the consensus features of the i-th image output by the group consensus aggregation module;

[0027] The attention calculation is specifically as follows:

[0028] After the feature map of a certain type of image is transformed, it is assigned to query Q, key K and value V respectively. Query Q is used to measure the degree of correlation between each feature in the non-local block and other features, key K is used to indicate the importance of each feature in the non-local block, and value V contains the specific information of each feature in the non-local block. The attention of the same type of image is calculated using the following formula:

[0029]

[0030] Where Attention(Q,K,V) represents the attention of the same type of images, which is used to indicate the features that should be paid attention to in this type of images; d represents the channel dimension, Softmax(·) represents the Softmax activation function, and T represents transpose.

[0031] Furthermore, the specific process of updating the consensus feature by the main branch memory comparison module and the consensus feature by the secondary branch memory comparison module is:

[0032] The memory comparison module contains the same number of memory blocks as the image categories. The consensus features of each category of images are saved and updated in their own memory blocks. The update rules are as follows:

[0033]

[0034] In the formula, represents the consensus feature of the i-th image at time t, represents the consensus feature of the i-th image at time t-1, β is the momentum factor, represents the consensus features of the i-th image output by the group consensus aggregation module;

[0035] The memory comparison module is supervised by the triplet loss, which is expressed as follows:

[0036]

[0037] In the formula, represents the consensus feature of image type i at time t, C j t represents the consensus feature of image class j at time t,

[0038] represents triple loss, anchor represents anchor point, positive represents positive sample, negative represents negative sample, ||·||2 represents l2 norm, is the hyperparameter enforced between positive and negative samples in the triplet loss, L MCM Represents the average value of the triplet loss calculated for all categories of images in one training cycle; N represents the total number of image categories.

[0039] Furthermore, the adversarial integrity learning of the overall prediction graph is specifically as follows:

[0040] Multiply the true value label GT by the image group G to get the true value map G GT , the formula is as follows:

[0041] G GT =GT·G

[0042] Multiply the overall prediction map M output by the main branch decoder with the image group G to obtain the mask segmentation map G of the overall prediction map M , the formula is as follows:

[0043] G M =M·G

[0044] Consider the main branch as a generator, and then construct a discriminator to judge the truth value graph G GT And the mask segmentation map G of the overall prediction map M The discriminator is used to guide the generator to adjust the generated overall prediction map;

[0045] The loss function of the discriminator is as follows:

[0046]

[0047] T=discriminator(G GT ·G)

[0048]

[0049] L BCE =-∑[G GT log(G M ),(1-G GT )log(1-G M )]

[0050]

[0051] In the formula, Represents the mask segmentation map G of the discriminator for the overall prediction map M The true or false judgment result, T represents the discriminator for the truth value graph G GT The true or false judgment result, discriminator (·) represents the discriminator, which consists of a linear layer and a Sigmoid function, L adv Indicates the judgment result and the binary cross entropy loss of T, L BCE The mask segmentation map G representing the overall prediction map M and truth graph G GT The binary cross entropy loss, L IoUThe mask segmentation map G representing the overall prediction map M and truth graph G GT The intersection-over-union loss is , and N represents the total category of the image.

[0052] Furthermore, the knowledge distillation of the fusion prediction graph is specifically as follows:

[0053] The encoder, main branch and auxiliary branch are considered as a student model, and the fusion prediction graph M s is regarded as the output of the student model,

[0054] The pre-trained teacher model generates weak labels W corresponding to the image group G l ;

[0055] Using fusion prediction graph M s With weak label W l The binary cross entropy loss and intersection-over-union loss are calculated to guide the student model to learn the predictive ability of the teacher model so that the student model outputs a co-saliency prediction map.

[0056] The present invention also proposes a collaborative salient target detection system based on multi-target constraints, comprising:

[0057] The encoder is used to extract the features of the image group and obtain a feature map;

[0058] A parallel branch learning module is used to perform parallel branch learning on the feature map. The parallel branch learning module includes a main branch and a sub-branch. The main branch is responsible for learning the overall content of the object in the image to obtain the overall prediction map, and the sub-branch is responsible for learning the boundary content of the object in the image to obtain the boundary prediction map;

[0059] The adversarial integrity learning module is used to perform adversarial integrity learning on the overall prediction graph and guide the main branch to adjust the overall prediction graph;

[0060] A fusion module, used for fusing the boundary prediction map with the adjusted overall prediction map to obtain a fused prediction map;

[0061] The knowledge distillation module is used to perform knowledge distillation on the fusion prediction map to obtain the co-saliency prediction map.

[0062] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the collaborative salient target detection method based on multi-target constraints as described above is implemented.

[0063] The present invention also proposes a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the collaborative salient target detection method based on multi-target constraints as described above.

[0064] The beneficial effects of the present invention are:

[0065] 1. The present invention adopts (Pyramid Vision Transformerv2) PVTv2 for feature encoding. The network can better capture information of different scales in the image, improve the ability to understand complex images, and can be easily combined with other model structures, so that it can perform well in processing various types of image tasks.

[0066] 2. The present invention adopts a two-branch parallel learning method, the main branch is responsible for learning the overall content of the object, and the secondary branch is responsible for learning the boundary content of the object. For the case where the details of the prediction image are blurred, missing, and the edge confidence is low, an additional branch is introduced to be responsible for learning the edge content of the object, and finally interacts with the overall content, which can make fuller use of the feature information and contribute to the quality of the final prediction image.

[0067] 3. Using the attention mechanism in the group consensus aggregation module, the features within the block are weighted in the attention mechanism, which can better capture the relationship between the features within the block.

[0068] 4. Using the knowledge distillation method, through the guidance of teacher models with different prediction capabilities, the student model will have a balanced improvement in the detection ability of different targets in different scenarios. At the same time, absorbing the knowledge of different teachers can enable the student model to have detection capabilities that exceed the teacher. Accelerate the learning of simple samples, avoid overfitting to difficult samples, and integrate different knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 The overall structure diagram of the network for realizing a collaborative salient target detection method based on multi-target constraints proposed by the present invention;

[0070] Figure 2 This is the knowledge distillation structure diagram;

[0071] Figure 3 A schematic diagram of the test results. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0073] Embodiment 1

[0074] The present invention proposes a collaborative salient target detection method based on multi-target constraints. The overall structure diagram of the network implementing this method is as follows: Figure 1 As shown, the method comprises the following steps:

[0075] The image group is input into the encoder to extract features and obtain feature maps; the image group consists of N different types of images {G1,…,G i ,…,G N} is stacked in the batch dimension, G i represents the i-th image; the annotation set corresponding to the image group G is a series of binary segmentation images, namely, the true value label GT. In this embodiment, the input image group is two types of images, one is airplanes and the other is dogs.

[0076] In this embodiment, the encoder uses Pyramid Vision Transformer v2, and the image group G is input into the pre-trained Pyramid Vision Transformer v2 for feature extraction. Pyramid Vision Transformer v2 has 4 network stages, and each network stage generates feature maps of different scales. represents the feature map output by the first network stage of the encoder, represents the feature map output by the second network stage of the encoder, represents the feature map output by the third network stage of the encoder, Feature map representing the output of the fourth network stage of the encoder.

[0077] The feature map is learned in parallel. The main branch is responsible for learning the overall content of the object in the image to obtain the overall prediction map, and the secondary branch is responsible for learning the boundary content of the object in the image to obtain the boundary prediction map.

[0078] The specific steps of parallel branch learning are as follows:

[0079] The main branch includes a main branch convolutional layer, a main branch group consensus aggregation module (GCAM), a main branch memory comparison module, and a main branch decoder;

[0080] The side branch includes a side branch convolution layer, a side branch group consensus aggregation module, a side branch memory comparison module and a side branch decoder;

[0081] The main branch decoder has 4 network stages, and the feature map output by the first network stage of the encoder is Feature map output from the second network stage of the encoder Feature map output by the third network stage of the encoder The feature graph output by the main branch convolution layer is convolved and added with the network stage corresponding to the main branch decoder;

[0082] The sub-branch decoder has 4 network stages, and the feature map output by the first network stage of the encoder is Feature map output from the second network stage of the encoder Feature map output by the third network stage of the encoder The feature graph output by the sub-branch convolution layer is convolved and added with the network stage corresponding to the sub-branch decoder;

[0083] Feature map output by the fourth network stage of the encoder The consensus features are extracted by the main branch group consensus aggregation module, and then updated by the main branch memory comparison module and input into the main branch decoder to generate the overall prediction graph;

[0084] Feature map output by the fourth network stage of the encoder The consensus features are extracted by the sub-branch group consensus aggregation module, and then updated by the sub-branch memory comparison module and input into the sub-branch decoder to generate a boundary prediction map.

[0085] The specific process of extracting consensus features by the main branch group consensus aggregation module and the sub-branch group consensus aggregation module is as follows:

[0086] The feature map output by the fourth network stage of the encoder In the batch dimension, we get Represents the feature map of class i image, Attention calculation is performed in the non-local block of the input group consensus aggregation module, specifically:

[0087] After the feature map of a certain type of image is transformed, it is assigned to query Q, key K and value V respectively. Query Q is used to measure the degree of correlation between each feature in the non-local block and other features, key K is used to indicate the importance of each feature in the non-local block, and value V contains the specific information of each feature in the non-local block. The attention of the same type of image is calculated using the following formula:

[0088]

[0089] Where Attention(Q,K,V) represents the attention of the same type of images, which is used to indicate the features that should be paid attention to in this type of images; d represents the channel dimension, Softmax(·) represents the Softmax activation function, and T represents transpose.

[0090] The output of the non-local block is recorded as Represents the consensus feature of class i image, which is the same as the input After corresponding multiplication, the consensus features output by the final group consensus aggregation module are obtained by re-joining them in the batch dimension. The specific expression is:

[0091]

[0092] In the formula, concatenate(·) represents the concatenation operation, and n is the batch size; represents the consensus features of the i-th image output by the group consensus aggregation module;

[0093] The specific process of updating the consensus feature of the main branch memory comparison module and the consensus feature of the secondary branch memory comparison module is:

[0094] The memory comparison module contains the same number of memory blocks as the image categories. The consensus features of each category of images are saved and updated in their own memory blocks. The update rules are as follows:

[0095]

[0096] In the formula, represents the consensus feature of the i-th image at time t, represents the consensus feature of the i-th image at time t-1, β is the momentum factor, which is set to 0.1 by default. represents the consensus features of the i-th image output by the group consensus aggregation module;

[0097] The memory comparison module is supervised by triplet loss, which is expressed as follows:

[0098]

[0099] In the formula, represents the consensus feature of image type i at time t, C j t represents the consensus feature of image class j at time t, represents triple loss, anchor represents anchor point, positive represents positive sample, negative represents negative sample, ||·||2 represents l2 norm, is a hyperparameter that is forced to be set between positive and negative samples in the triple loss, and is set to 0.1 by default. MCM It represents the average value of triplet loss calculated for all categories of images in a training cycle, which can enhance the uniqueness and difference of consensus features of objects of different categories. N represents the total categories of images.

[0100] Perform adversarial integrity learning (AIL) on the overall prediction graph to guide the main branch to adjust the overall prediction graph; specifically:

[0101] Multiply the true value label GT by the image group G to get the true value map G GT Obviously G GT With 100% completeness, the formula is as follows:

[0102] G GT =GT·G

[0103] Multiply the overall prediction map M output by the main branch decoder with the image group G to obtain the mask segmentation map G of the overall prediction map M , the formula is as follows:

[0104] G M =M·G

[0105] Consider the main branch as a generator, and then construct a discriminator to judge the truth value graph G GT And the mask segmentation map G of the overall prediction map M The discriminator is used to guide the generator to adjust the generated overall prediction graph so that it can generate prediction results with higher accuracy and completeness.

[0106] The loss function of the discriminator is as follows:

[0107]

[0108] T=discriminator(G GT ·G)

[0109]

[0110] L BCE =-∑[G GT log(G M ),(1-G GT )log(1-G M )]

[0111]

[0112] In the formula, Represents the mask segmentation map G of the discriminator for the overall prediction map M The true or false judgment result, T represents the discriminator for the truth value graph G GT The true or false judgment result, discriminator (·) represents the discriminator, which consists of a linear layer and a Sigmoid function, L adv Indicates the judgment result and the binary cross entropy loss of T, L BCE The mask segmentation map G representing the overall prediction map M and truth graph G GT The binary cross entropy loss can improve the integrity of the overall prediction graph M, LIoU The mask segmentation map G representing the overall prediction map M and truth graph G GT The intersection-over-union loss can improve the confidence of the final result, and N represents the total category of the image.

[0113] The boundary prediction map is fused with the adjusted overall prediction map to obtain a fused prediction map;

[0114] The fusion prediction graph is subjected to knowledge distillation to obtain the co-saliency prediction graph. The process of knowledge distillation is as follows: Figure 2 As shown, specifically:

[0115] The encoder, main branch and auxiliary branch are considered as a student model, and the fusion prediction graph M s is regarded as the output of the student model,

[0116] The pre-trained teacher model generates weak labels W corresponding to the image group G l ;

[0117] Using fusion prediction graph M s With weak label W l The binary cross entropy loss and intersection-over-union loss are calculated to guide the student model to learn the predictive ability of the teacher model so that the student model outputs a co-saliency prediction map.

[0118] The calculation formulas for binary cross entropy loss and intersection-over-union loss are as follows:

[0119] L BcE ′=-∑[W l log(M s ),(1-W l )log(1-M s )]

[0120]

[0121] Where, L BCE ′ represents the binary cross entropy loss between the teacher prediction graph and the student prediction graph, L IoU ′ represents the intersection-over-union loss of the teacher prediction graph and the student prediction graph. The combination of the two can guide the student model to learn the detection ability of the teacher model. N represents the total category of the image.

[0122] Embodiment 2

[0123] The present invention proposes a collaborative salient object detection system based on multi-objective constraints corresponding to the method of embodiment 1, comprising:

[0124] The encoder is used to extract the features of the image group and obtain a feature map;

[0125] A parallel branch learning module is used to perform parallel branch learning on the feature map. The parallel branch learning module includes a main branch and a sub-branch. The main branch is responsible for learning the overall content of the object in the image to obtain the overall prediction map, and the sub-branch is responsible for learning the boundary content of the object in the image to obtain the boundary prediction map;

[0126] The adversarial integrity learning module is used to perform adversarial integrity learning on the overall prediction graph and guide the main branch to adjust the overall prediction graph;

[0127] A fusion module, used for fusing the boundary prediction map with the adjusted overall prediction map to obtain a fused prediction map;

[0128] The knowledge distillation module is used to perform knowledge distillation on the fusion prediction map to obtain the co-saliency prediction map.

[0129] The implementation methods of each module and module function in the system are completely consistent with the steps of the method in Example 1, so they will not be repeated here.

[0130] Embodiment 3

[0131] The present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the collaborative salient target detection method based on multi-target constraints as described in Example 1 is implemented.

[0132] Embodiment 4

[0133] The present invention proposes a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the collaborative salient target detection method based on multi-target constraints as described in Example 1.

[0134] The effect of the present invention is described below in conjunction with experiments.

[0135] NVIDIA GPU graphics card is used as the computing platform, and Pytorch is used as the network model framework.

[0136] 1. Dataset acquisition and introduction

[0137] Download the training sets DUTS_class and COCO-SEG, and the test sets CoCA, CoSOD3k, and CoSal2015 from the Internet. DUTS_class has 8250 images in 291 categories, and COCO-SEG has 200k images in 78 categories.

[0138] 2. Model training

[0139] In order to solve the two subtasks of CoSOD, namely common object segmentation and salient object detection, this paper needs to optimize the student model in two directions. Therefore, this paper sets up joint training and jointly trains the student model on DUTS_class and COCO-SEG. Under the design of joint training, the model shows more robust performance. The batch size is set to 16.

[0140] The input images are resized to 256x256 for both training and inference. The co-saliency prediction maps are resized to their original size for evaluation. We employ three data augmentation strategies, namely horizontal flipping, color augmentation, and rotation. Our model is trained for 300 epochs using the AdamW optimizer with an initial learning rate of 1e-4, which is divided by 10 in the last 20 epochs.

[0141] The trained model is used for the detection results of co-saliency detection. Figure 3 shown.

[0142] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0143] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0144] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A collaborative salient object detection method based on multi-objective constraints, characterized in that: The following steps are involved: Input the image group into the encoder to extract features and obtain a feature map; The feature map is learned in parallel. The main branch is responsible for learning the overall content of the object in the image to obtain the overall prediction map, and the secondary branch is responsible for learning the boundary content of the object in the image to obtain the boundary prediction map. Conduct adversarial integrity learning on the overall prediction graph to guide the main branch to adjust the overall prediction graph; The boundary prediction map is fused with the adjusted overall prediction map to obtain a fused prediction map; Knowledge distillation is performed on the fused prediction map to obtain the co-saliency prediction map.

2. The collaborative salient target detection method based on multi-target constraints according to claim 1, characterized in that: The image group consists of N different types of images {G1, ..., G i , ..., G N } is stacked in the batch dimension, G i represents the i-th category image; The encoder is Pyramid Vision Transformer v2. The image group G is input into the pre-trained Pyramid Vision Transformer v2 for feature extraction. Pyramid Vision Transformer v2 has 4 network stages, and each network stage generates feature maps of different scales. represents the feature map output by the first network stage of the encoder, represents the feature map output by the second network stage of the encoder, represents the feature map output by the third network stage of the encoder, Feature map representing the output of the fourth network stage of the encoder.

3. The collaborative salient target detection method based on multi-target constraints according to claim 2, characterized in that: The parallel branch learning is specifically as follows: The main branch includes a main branch convolution layer, a main branch group consensus aggregation module, a main branch memory comparison module and a main branch decoder; The side branch includes a side branch convolution layer, a side branch group consensus aggregation module, a side branch memory comparison module and a side branch decoder; The main branch decoder has 4 network stages, and the feature map output by the first network stage of the encoder is Feature map output from the second network stage of the encoder Feature map output by the third network stage of the encoder The feature graph output by the main branch convolution layer is convolved and added with the network stage corresponding to the main branch decoder; The sub-branch decoder has 4 network stages, and the feature map output by the first network stage of the encoder is Feature map output from the second network stage of the encoder Feature map output by the third network stage of the encoder The feature graph output by the sub-branch convolution layer is convolved and added with the network stage corresponding to the sub-branch decoder; Feature map output by the fourth network stage of the encoder The consensus features are extracted by the main branch group consensus aggregation module, and then updated by the main branch memory comparison module and input into the main branch decoder to generate the overall prediction graph; Feature map output by the fourth network stage of the encoder The consensus features are extracted by the sub-branch group consensus aggregation module, and then updated by the sub-branch memory comparison module and input into the sub-branch decoder to generate a boundary prediction map.

4. The collaborative salient target detection method based on multi-target constraints according to claim 3, characterized in that: The specific process of extracting consensus features by the main branch group consensus aggregation module and the sub-branch group consensus aggregation module is as follows: The feature map output by the fourth network stage of the encoder In the batch dimension, we get Represents the feature map of class i image, The attention calculation is performed in the non-local block of the input group consensus aggregation module, and the output of the non-local block is recorded as Represents the consensus feature of class i image, which is the same as the input After corresponding multiplication, the consensus features output by the final group consensus aggregation module are obtained by re-joining them in the batch dimension. The specific expression is: In the formula, concatenate(·) represents the concatenation operation, and n is the batch size; represents the consensus features of the i-th image output by the group consensus aggregation module; The attention calculation is specifically as follows: After the feature map of a certain type of image is transformed, it is assigned to query Q, key K and value V respectively. Query Q is used to measure the degree of correlation between each feature in the non-local block and other features, key K is used to indicate the importance of each feature in the non-local block, and value V contains the specific information of each feature in the non-local block. The attention of the same type of image is calculated using the following formula: In the formula, Attention(Q, K, V) represents the attention of the same type of images, which is used to indicate the features that should be paid attention to in this type of images; d represents the channel dimension, Softmax(·) represents the Softmax activation function, and T represents transpose.

5. The collaborative salient target detection method based on multi-target constraints according to claim 3, characterized in that: The specific process of updating the consensus feature of the main branch memory comparison module and the consensus feature of the secondary branch memory comparison module is: The memory comparison module contains the same number of memory blocks as the image categories. The consensus features of each category of images are saved and updated in their own memory blocks. The update rules are as follows: In the formula, represents the consensus feature of the i-th image at time t, represents the consensus feature of the i-th image at time t-1, β is the momentum factor, represents the consensus features of the i-th image output by the group consensus aggregation module; The memory comparison module is supervised by the triplet loss, which is expressed as follows: In the formula, represents the consensus feature of image type i at time t, C j t represents the consensus feature of image class j at time t, represents triple loss, anchor represents anchor point, positive represents positive sample, negative represents negative sample, ||·||2 represents l2 norm, is the hyperparameter enforced between positive and negative samples in the triplet loss, L MCM Represents the average value of the triplet loss calculated for all categories of images in one training cycle; N represents the total number of image categories.

6. The collaborative salient target detection method based on multi-target constraints according to claim 3, characterized in that: The adversarial integrity learning of the overall prediction graph is specifically as follows: Multiply the true value label GT by the image group G to get the true value map G cT , the formula is as follows: G GT =GT·G Multiply the overall prediction map M output by the main branch decoder with the image group G to obtain the mask segmentation map G of the overall prediction map M , the formula is as follows: G M =M·G Consider the main branch as a generator, and then construct a discriminator to judge the truth value graph G GT And the mask segmentation map G of the overall prediction map M The discriminator is used to guide the generator to adjust the generated overall prediction map; The loss function of the discriminator is as follows: T=discriminator(G GT ·G) L BCE =-∑[G GT log(G M ),(1-G GT )log(1-G M )] In the formula, Represents the mask segmentation map G of the discriminator for the overall prediction map M The true or false judgment result, T represents the discriminator for the truth value graph G GT The true or false judgment result, discriminator (·) represents the discriminator, which consists of a linear layer and a Sigmoid function, L adv Indicates the judgment result and the binary cross entropy loss of T, L BCE The mask segmentation map G representing the overall prediction map M and truth graph G GT The binary cross entropy loss, L IoU The mask segmentation map G representing the overall prediction map M and truth graph G GT The intersection-over-union loss is , and N represents the total category of the image.

7. The collaborative salient target detection method based on multi-target constraints according to claim 1, characterized in that: The knowledge distillation of the fusion prediction graph is specifically as follows: The encoder, main branch and auxiliary branch are considered as a student model, and the fusion prediction graph M s is regarded as the output of the student model, The pre-trained teacher model generates weak labels W corresponding to the image group G l ; Using fusion prediction graph M s With weak label W l The binary cross entropy loss and intersection-over-union loss are calculated to guide the student model to learn the predictive ability of the teacher model so that the student model outputs a co-saliency prediction map.

8. A collaborative salient object detection system based on multi-objective constraints, characterized in that: include: The encoder is used to extract the features of the image group and obtain a feature map; A parallel branch learning module is used to perform parallel branch learning on the feature map. The parallel branch learning module includes a main branch and a sub-branch. The main branch is responsible for learning the overall content of the object in the image to obtain the overall prediction map, and the sub-branch is responsible for learning the boundary content of the object in the image to obtain the boundary prediction map; The adversarial integrity learning module is used to perform adversarial integrity learning on the overall prediction graph and guide the main branch to adjust the overall prediction graph; A fusion module, used for fusing the boundary prediction map with the adjusted overall prediction map to obtain a fused prediction map; The knowledge distillation module is used to perform knowledge distillation on the fusion prediction map to obtain the co-saliency prediction map.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the collaborative salient target detection method based on multi-target constraints as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to execute the collaborative salient target detection method based on multi-target constraints as described in any one of claims 1 to 7.

Images