Collaborative salient object detection method, system, electronic device and storage medium

Abstract

The present application discloses a collaborative salient object detection method, system, electronic device and storage medium, and the technical field to which it belongs is image processing technology. The collaborative salient object detection method includes: dividing the images in the first image group and the second image group into image blocks, obtaining and using the T2T structure to perform dimensionality reduction processing on and and fuse local information, obtaining and inserting corresponding category primitives in and respectively, and using the Transformer structure to perform feature fusion operations, deep feature extraction operations and global information fusion operations on and; removing the category primitives in the deep-level fusion features, obtaining and using similarity loss and first cross entropy loss to adjust the model parameters of the dual-branch collaborative saliency detection model; using the dual-branch collaborative saliency detection model to detect collaborative saliency objects. The present application can improve the detection accuracy of collaborative saliency objects.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a collaborative salient object detection method, system, electronic device and storage medium. Background Art

[0002] Related technologies usually implement co-salient object detection based on deep convolutional neural networks, which guide the model to detect and segment co-salient targets with the same semantics by exploring the deep features of the image. However, the above methods generally only extract the common features of a group of images, and then fuse the common features with the features of each image, thereby ignoring the use of inter-group information. Moreover, these methods are all based on convolutional neural networks, and the receptive field of convolutional neural networks is limited. Therefore, the extracted features have low detection accuracy when facing complex scenes with more interference.

[0003] Therefore, how to improve the detection accuracy of co-salient objects is a technical problem that those skilled in the art currently need to solve. Summary of the invention

[0004] The purpose of this application is to provide a collaborative salient object detection method, system, electronic device and storage medium to solve the technical problem of how to improve the detection accuracy of collaborative salient objects.

[0005] In order to solve the above technical problems, the present application provides a collaborative salient object detection method, which is applied to a device running a dual-branch collaborative salient detection model, wherein the dual-branch collaborative salient detection model includes a Tokens-to-Token structure, a Transformer structure, an auxiliary classifier, and a Brownian distance covariance calculation module. The collaborative salient object detection method includes:

[0006] Divide the images in the first image group and the second image group into image blocks to obtain a first image element sequence and the second primitive sequence

[0007] The first primitive sequence is coded using the Tokens-to-Token structure. and the second primitive sequence Perform dimensionality reduction and fuse local information to obtain the first primitive sequence and the second primitive sequence

[0008] In the first primitive sequence and the second primitive sequence Insert the corresponding category primitives into each, and use the Transformer structure to transform the first primitive sequence and the second primitive sequence Perform feature fusion operations, deep feature extraction operations and global information fusion operations to obtain deep-level fusion features;

[0009] Remove the category primitives in the deep fusion feature to obtain the first primitive sequence and the second primitive sequence

[0010] The first primitive sequence is calculated using the Brownian distance covariance calculation module and the second primitive sequence similarity information, and calculating a similarity loss according to the similarity information;

[0011] The first primitive sequence and the second primitive sequence Input the auxiliary classifier to obtain a category prediction result, and calculate a first cross entropy loss according to the category prediction result;

[0012] Adjusting model parameters of the dual-branch collaborative saliency detection model using the similarity loss and the first cross entropy loss;

[0013] Receiving a collaborative salient object detection task, and determining a class A image group and a class B image group corresponding to the collaborative salient object detection task;

[0014] The dual-branch co-saliency detection model is used to detect co-saliency objects in the A-type image group and the B-type image group.

[0015] Optionally, after removing the category primitives in the deep fusion feature, a first primitive sequence is obtained. and the second primitive sequence After that, it also includes:

[0016] For the first primitive sequence and the second primitive sequence Perform upsampling and MLP operations to obtain the first primitive sequence and the second primitive sequence

[0017] In the first primitive sequence and the second primitive sequence Insert the corresponding category of graphics elements into the first graphics element sequence and the second primitive sequence Obtain a first spliced ​​primitive sequence, use a Transformer structure to interact and fuse the first spliced ​​primitive sequence, remove the category primitives, and obtain a fused primitive sequence

[0018] The fused primitive sequence is decoded by a decoder Performing collaborative salient target prediction to obtain a first predicted mask map, and calculating a second cross entropy loss and an IoU loss between the first predicted mask map and the corresponding label;

[0019] Accordingly, adjusting the model parameters of the dual-branch collaborative saliency detection model by using the similarity loss and the first cross entropy loss includes:

[0020] A total loss is calculated according to the similarity loss, the first cross entropy loss, the second cross entropy loss and the IoU loss, and a model parameter of the dual-branch collaborative saliency detection model is adjusted using the total loss.

[0021] Optionally, using the dual-branch co-saliency detection model to detect co-saliency objects in the class A image group and the class B image group includes:

[0022] The images in the A-type image group and the B-type image group are divided into image blocks to obtain a sequence of A-type image elements. and B-type primitive sequence

[0023] The A-type primitive sequence is processed by using the Tokens-to-Token structure. and the B-type primitive sequence Perform dimensionality reduction and fuse local information to obtain a sequence of type A graphics elements and B-type primitive sequence

[0024] In the A-type primitive sequence and the B-type primitive sequence The corresponding category primitives are inserted into the A-category primitives, and the Transformer structure is used to transform the A-category primitive sequence and the B-type primitive sequence Perform feature fusion operations, deep feature extraction operations and global information fusion operations to obtain deep-level fusion features;

[0025] Remove the category primitives in the deep fusion feature to obtain a class A primitive sequence and B-type primitive sequence

[0026] For the A-type primitive sequence and B-type primitive sequence Perform upsampling and MLP operations to obtain a sequence of type A graph elements. and B-type primitive sequence

[0027] In the A-type primitive sequence and the B-type primitive sequence Insert the corresponding category of graphics elements into the A-type graphics element sequence and the B-type primitive sequence Obtain a second spliced ​​primitive sequence, interact and fuse the second spliced ​​primitive sequence using a Transformer structure, remove the category primitives, and obtain a fused primitive sequence X4;

[0028] The decoder is used to perform co-salient target prediction on the fused primitive sequence X4 to obtain a second prediction mask map, and the co-salient objects in the class A image group and the class B image group are determined according to the second prediction mask map.

[0029] Optionally, concatenate the A-type graphic element sequence and the B-type primitive sequence The second stitching primitive sequence is obtained, including:

[0030] The A-type primitive sequence and the B-type primitive sequence The stitching is performed along the spatial dimension to obtain the second stitching primitive sequence.

[0031] Optionally, determining the co-salient objects in the class A image group and the class B image group according to the second prediction mask map includes:

[0032] The white area in the second prediction mask image is used as the co-salient object in the A-type image group and the B-type image group.

[0033] Optionally, dividing the images in the first image group and the second image group into image blocks includes:

[0034] The images in the first image group and the second image group are divided into image blocks using sliding windows with overlap.

[0035] Optionally, the first primitive sequence is processed by using a Tokens-to-Token structure. and the second primitive sequence Perform dimensionality reduction and fuse local information, including:

[0036] The first primitive sequence and the second primitive sequence The input is to the two Tokens-to-Token structures connected in series to perform dimensionality reduction and fuse local information.

[0037] Optionally, the Brownian distance covariance calculation module is used to calculate the first primitive sequence and the second primitive sequence Similarity information includes:

[0038] The first primitive sequence is calculated using the Brownian distance covariance calculation module The corresponding intra-group similarity information;

[0039] The second primitive sequence is calculated using the Brownian distance covariance calculation module The corresponding intra-group similarity information;

[0040] The first primitive sequence is calculated using the Brownian distance covariance calculation module With the second primitive sequence The inter-group similarity information.

[0041] The present application also provides a collaborative salient object detection system, which is applied to a device running a dual-branch collaborative salient detection model, wherein the dual-branch collaborative salient detection model includes a Tokens-to-Token structure, a Transformer structure, an auxiliary classifier, and a Brownian distance covariance calculation module, and the collaborative salient object detection system includes:

[0042] An image division module is used to divide the images in the first image group and the second image group into image blocks to obtain a first image element sequence and the second primitive sequence

[0043] A dimension reduction module is used to reduce the first primitive sequence using a Tokens-to-Token structure. and the second primitive sequence Perform dimensionality reduction and fuse local information to obtain the first primitive sequence and the second primitive sequence

[0044] A fusion module is used to combine the first primitive sequence and the second primitive sequence Insert the corresponding category primitives into each, and use the Transformer structure to transform the first primitive sequence and the second primitive sequence Perform feature fusion operations, deep feature extraction operations and global information fusion operations to obtain deep-level fusion features;

[0045] The primitive processing module is used to remove the category primitives in the deep fusion features to obtain a first primitive sequence and the second primitive sequence

[0046] A parameter adjustment module, used to calculate the first primitive sequence using the Brownian distance covariance calculation module and the second primitive sequence similarity information, and calculating the similarity loss according to the similarity information; and also used to convert the first primitive sequence and the second primitive sequence Inputting the auxiliary classifier to obtain a category prediction result, and calculating a first cross entropy loss according to the category prediction result; and also used to adjust the model parameters of the dual-branch collaborative saliency detection model using the similarity loss and the first cross entropy loss;

[0047] A task receiving module, used to receive a collaborative salient object detection task and determine a class A image group and a class B image group corresponding to the collaborative salient object detection task;

[0048] The task processing module is used to detect the co-salient objects in the A-type image group and the B-type image group using the dual-branch co-saliency detection model.

[0049] The present application also provides a storage medium on which a computer program is stored. When the computer program is executed, the steps of the collaborative salient object detection method are implemented.

[0050] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the steps of the collaborative salient object detection method are implemented.

[0051] The present application provides a collaborative salient object detection method, which is applied to a device running a dual-branch collaborative salient detection model, wherein the dual-branch collaborative salient detection model includes a Tokens-to-Token structure, a Transformer structure, an auxiliary classifier, and a Brownian distance covariance calculation module. The collaborative salient object detection method includes: dividing images in a first image group and a second image group into image blocks to obtain a first primitive sequence and the second primitive sequence The first primitive sequence is coded using the Tokens-to-Token structure. and the second primitive sequence Perform dimensionality reduction and fuse local information to obtain the first primitive sequence and the second primitive sequence In the first primitive sequence and the second primitive sequence Insert the corresponding category primitives into each, and use the Transformer structure to transform the first primitive sequence and the second primitive sequence Perform feature fusion operation, deep feature extraction operation and global information fusion operation to obtain deep-level fusion features; remove the category primitives in the deep-level fusion features to obtain a first primitive sequence and the second primitive sequence The first primitive sequence is calculated using the Brownian distance covariance calculation module and the second primitive sequence similarity information of the first primitive sequence, and calculating the similarity loss according to the similarity information; and the second primitive sequence The auxiliary classifier is input to obtain a category prediction result, and a first cross entropy loss is calculated according to the category prediction result; the model parameters of the dual-branch collaborative saliency detection model are adjusted using the similarity loss and the first cross entropy loss; a collaborative saliency object detection task is received, and a class A image group and a class B image group corresponding to the collaborative saliency object detection task are determined; and the dual-branch collaborative saliency detection model is used to detect collaborative saliency objects in the class A image group and the class B image group.

[0052] The present application performs image block division on the images in the first image group and the second image group to obtain the first primitive sequence and the second primitive sequence, and uses the Tokens-to-Token structure to perform dimensionality reduction processing and local feature fusion on the first primitive sequence and the second primitive sequence. The present application also uses the Transformer structure to perform feature fusion operations, deep feature extraction operations and global information fusion operations on the first primitive sequence and the second primitive sequence, and uses the Brownian distance covariance calculation module to calculate the similarity within and between image groups and determine the similarity loss based on the calculated similarity information. The present application also uses an auxiliary classifier to calculate the first cross entropy loss, and uses the similarity loss and the first cross entropy loss to adjust the model parameters of the dual-branch collaborative saliency detection model, and then uses the dual-branch collaborative saliency detection model after adjusting the parameters to detect collaborative saliency objects. Therefore, the present application can improve the detection accuracy of collaborative saliency objects. The present application also provides a collaborative saliency object detection system, a storage medium and an electronic device, which have the above-mentioned beneficial effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1A flowchart of a collaborative salient object detection method provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of the workflow of a dual-branch collaborative saliency detection model provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of a T2T structure provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of a Transformer structure provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of the results of a dual-branch collaborative saliency detection based on Brownian distance provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] The goal of co-saliency detection is to detect and segment co-saliency objects in a group of images, while the saliency detection task only needs to detect salient objects in one image. Therefore, co-saliency detection needs to extract target features of a group of images at the same time, and needs to face more interfering objects and more complex backgrounds, and each target has different appearance features. Since this task has the ability to process multiple images and detect objects with the same semantic features, the development of this task is conducive to the development of co-segmentation, video foreground detection, video target tracking, image content understanding, etc.

[0061] See below Figure 1 , Figure 1 A flowchart of a collaborative salient object detection method provided in an embodiment of the present application.

[0062] Specific steps may include:

[0063] S101: Divide the images in the first image group and the second image group into image blocks to obtain a first image element sequence and the second primitive sequence

[0064] Among them, this embodiment can be applied to a device running a dual-branch collaborative saliency detection model, wherein the dual-branch collaborative saliency detection model includes a Tokens-to-Token (T2T) structure, a Transformer structure, an auxiliary classifier, and a Brownian distance covariance calculation module (i.e., a BDC module). The above-mentioned Transformer structure includes a multi-head self-attention layer, a feedforward network, a residual connection, and a layer regularization.

[0065] The dual-branch collaborative saliency detection model includes a first branch and a second branch for processing a first image group and a second image group respectively, and the first branch and the second branch include an input layer, an embedding layer, a T2T structure, a Transformer layer, an auxiliary classifier, a BDC module, an upsampling, and an MLP layer connected in sequence.

[0066] In this step, the two image groups can be input into the two branches of the network respectively, and then the pictures can be divided into multiple image blocks to form a primitive sequence. Specifically, the first image group can enter the dual-branch collaborative saliency detection model from the input layer of the first branch, and the image blocks are divided in the embedding layer of the first branch to obtain the first primitive sequence The second image group can enter the dual-branch collaborative saliency detection model from the input layer of the second branch, and perform image block division in the embedding layer of the second branch to obtain the second primitive sequence

[0067] S102: Utilizing a Tokens-to-Token structure to process the first primitive sequence and the second primitive sequence Perform dimensionality reduction and fuse local information to obtain the first primitive sequence and the second primitive sequence

[0068] In this embodiment, the first primitive sequence can be used and the second primitive sequence As input, each of them passes through n Tokens-to-Token structures to reduce the dimension of the primitive sequence, reduce the amount of calculation, and fuse local information to obtain the first primitive sequence and the second primitive sequence The preferred value of n is 2.

[0069] Specifically, the first primitive sequence can be Input the Tokens-to-Token structure of the first branch to perform dimensionality reduction and fuse local information to obtain the first primitive sequence You can also add the second primitive sequence Input the Tokens-to-Token structure of the second branch to perform dimensionality reduction and fuse local information to obtain the second primitive sequence

[0070] S103: In the first primitive sequence and the second primitive sequence Insert the corresponding category primitives into each, and use the Transformer structure to transform the first primitive sequence and the second primitive sequence Perform feature fusion operations, deep feature extraction operations and global information fusion operations to obtain deep-level fusion features;

[0071] Among them, the first primitive sequence obtained by T2T and the second primitive sequence The corresponding category primitives are inserted into the first primitive sequence to supervise the features and obtain more discriminative features. Insert the corresponding first category element in the second element sequence The corresponding second category primitives are inserted into the . Category primitives refer to the class token in the ViT (Vision Transformer) model, which is used to supervise the category.

[0072] This step also takes the primitive sequence after adding the category primitives as input, uses a series of Transformer layers to perform feature fusion and deep feature extraction, and simultaneously fuses global information to obtain deep-level fusion features.

[0073] S104: removing the category primitives in the deep-level fusion features to obtain a first primitive sequence and the second primitive sequence

[0074] This step is used to remove the category primitives added in the previous step to obtain the first primitive sequence. and the second primitive sequence In this embodiment, the first primitive sequence and is a sequence of image primitives obtained after each image in the first image group is processed at different stages. In this embodiment, the second image primitive sequence and It is a sequence of picture elements obtained after each image in the second image group is processed at different stages.

[0075] The BDC module is used to establish a joint distribution for feature maps, placing features of different categories farther apart in the feature space and features of the same category closer together. Similarity information is established for all features of the image, while the spatial scale of the category primitive is 1, which only contains category information. Therefore, before the primitive sequence is input into the BDC module, the category primitive needs to be removed.

[0076] S105: Calculate the first primitive sequence using the Brownian distance covariance calculation module and the second primitive sequence similarity information, and calculating a similarity loss according to the similarity information;

[0077] S106: The first primitive sequence and the second primitive sequence Input the auxiliary classifier to obtain a category prediction result, and calculate a first cross entropy loss according to the category prediction result;

[0078] S107: adjusting model parameters of the dual-branch collaborative saliency detection model using the similarity loss and the first cross entropy loss;

[0079] The deep fusion features of the two branches are removed to obtain the primitive sequence Then it is sent to the BDC module, which uses the joint distribution to distribute the two primitive sequences. The nonlinear relationship is modeled, and then similarity information is calculated according to the modeling results, where the similarity information includes intra-group similarity and inter-group similarity of each of the first image group and the second image group.

[0080] After obtaining the similarity information, the similarity loss can be calculated based on the similarity information. and the second primitive sequence The auxiliary classifier is input to obtain a category prediction result, and a first cross entropy loss is calculated according to the category prediction result. The similarity loss and the first cross entropy loss are combined to perform back propagation and stochastic gradient descent algorithm calculations to adjust the model parameters of the dual-branch collaborative saliency detection model, thereby completing the training process of the dual-branch collaborative saliency detection model.

[0081] S108: receiving a collaborative salient object detection task, and determining a class A image group and a class B image group corresponding to the collaborative salient object detection task;

[0082] After obtaining the trained dual-branch collaborative saliency detection model, a collaborative saliency object detection task can be received, and then the class A image group and the class B image group corresponding to the collaborative saliency object detection task can be determined, so as to determine the collaborative saliency objects in the class A image group and the class B image group. The first image group and the second image group are image groups used to train the dual-branch collaborative saliency detection model, and the class A image group and the class B image group are image groups for actual application after the dual-branch collaborative saliency detection model is trained.

[0083] S109: Detecting co-salient objects in the A-type image group and the B-type image group using the dual-branch co-saliency detection model.

[0084] Among them, this embodiment can input the images in the class A image group and the class B image group into a dual-branch co-saliency detection model, and then obtain the image content of the co-saliency objects in the class A image group and the co-saliency objects in the class B image group.

[0085] This embodiment divides the images in the first image group and the second image group into image blocks to obtain the first primitive sequence and the second primitive sequence, and uses the Tokens-to-Token structure to perform dimensionality reduction processing and local feature fusion on the first primitive sequence and the second primitive sequence. This embodiment also uses the Transformer structure to perform feature fusion operations, deep feature extraction operations and global information fusion operations on the first primitive sequence and the second primitive sequence, and uses the Brownian distance covariance calculation module to calculate the similarity within and between image groups and determine the similarity loss based on the calculated similarity information. This embodiment also uses an auxiliary classifier to calculate the first cross entropy loss, and uses the similarity loss and the first cross entropy loss to adjust the model parameters of the dual-branch collaborative saliency detection model, and then uses the dual-branch collaborative saliency detection model after adjusting the parameters to detect collaborative saliency objects. Therefore, this embodiment can improve the detection accuracy of collaborative saliency objects.

[0086] As a feasible implementation, the above embodiment can adjust the model parameters of the dual-branch collaborative saliency detection model in the following manner: using the Brownian distance covariance calculation module to calculate the first primitive sequence and the second primitive sequence similarity information of the first primitive sequence, and calculating the similarity loss according to the similarity information; and the second primitive sequence Input the auxiliary classifier to obtain a category prediction result, calculate the first cross entropy loss according to the category prediction result; and the second primitive sequence Perform upsampling and MLP operations to obtain the first primitive sequence and the second primitive sequence In the first primitive sequence and the second primitive sequence Insert the corresponding category of graphics elements into the first graphics element sequence and the second primitive sequence Obtain a first spliced ​​primitive sequence, use a Transformer structure to interact and fuse the first spliced ​​primitive sequence, remove the category primitives, and obtain a fused primitive sequence The fused primitive sequence is decoded by a decoder Perform collaborative saliency target prediction to obtain a first predicted mask map, calculate the second cross entropy loss and IoU (overlap) loss between the first predicted mask map and the corresponding label; calculate the total loss based on the similarity loss, the first cross entropy loss, the second cross entropy loss and the IoU loss, and use the total loss to adjust the model parameters of the dual-branch collaborative saliency detection model. In this embodiment, each image in the image group has its corresponding label, and the first predicted mask map is a mask map predicted for each image. In this embodiment, the second cross entropy loss and IoU loss are calculated based on the first predicted mask map and the label corresponding to the same image.

[0087] As a feasible implementation, the images in the first image group and the second image group may be divided into image blocks using overlapping sliding windows. Each image block boundary uses not only the content of the image block but also the content of other adjacent image blocks.

[0088] As a feasible implementation, the first primitive sequence can be and the second primitive sequence The input is to the two Tokens-to-Token structures connected in series to perform dimensionality reduction and fuse local information.

[0089] As a feasible implementation method, the similarity information can be determined in the following manner: using the Brownian distance covariance calculation module to calculate the first primitive sequence Corresponding intra-group similarity information; using the Brown distance covariance calculation module to calculate the second primitive sequence Corresponding intra-group similarity information; using the Brown distance covariance calculation module to calculate the first primitive sequence With the second primitive sequence The inter-group similarity information.

[0090] As a feasible implementation manner, the process of detecting the co-salient objects in the A-type image group and the B-type image group by the dual-branch co-saliency detection model includes:

[0091] Step 1: Divide the images in the A-type image group and the B-type image group into image blocks to obtain a sequence of A-type image elements. and B-type primitive sequence

[0092] In this embodiment, the A-type primitive sequence and is a sequence of image elements obtained after each image in the A-type image group is processed at different stages. In this embodiment, the sequence of image elements of type B is and It is a sequence of picture elements obtained after each image in the B-type image group is processed at different stages.

[0093] Step 2: Use the Tokens-to-Token structure to and the B-type primitive sequence Perform dimensionality reduction and fuse local information to obtain a sequence of type A graphics elements and B-type primitive sequence

[0094] Step 3: In the A-type primitive sequence and the B-type primitive sequence The corresponding category primitives are inserted into the A-category primitives, and the Transformer structure is used to transform the A-category primitive sequence and the B-type primitive sequence Perform feature fusion operations, deep feature extraction operations and global information fusion operations to obtain deep-level fusion features;

[0095] Step 4: Remove the category primitives in the deep fusion feature to obtain a sequence of category A primitives and B-type primitive sequence

[0096] This step can remove the category primitives from the deep-level fusion features corresponding to the A-type image group, and can also remove the category primitives from the deep-level fusion features corresponding to the B-type image group.

[0097] Step 5: For the A-type primitive sequence and B-type primitive sequence Perform upsampling and MLP operations to obtain a sequence of type A graph elements. and B-type primitive sequence

[0098] Step 6: In the A-type primitive sequence and the B-type primitive sequence Insert the corresponding category of graphics elements into the A-type graphics element sequence and the B-type primitive sequence Obtain a second spliced ​​primitive sequence, interact and fuse the second spliced ​​primitive sequence using a Transformer structure, remove the category primitives, and obtain a fused primitive sequence X4;

[0099] This step is in the A-type element sequence and B-type primitive sequence After inserting the category primitives, splicing is performed, and the Transformer layer is used to interact and fuse information, and after removing the category primitives, the fused primitive sequence X4 is obtained. Specifically, in this step, the A category primitive sequence and the B-type primitive sequence The second stitching primitive sequence is obtained by splicing along the spatial dimension. By inserting the column-specific primitives again, multiple fusion features can be achieved, so that the image features have more category information.

[0100] Step 7: Use a decoder to perform co-salient target prediction on the fused primitive sequence X4 to obtain a second prediction mask map, and determine the co-salient objects in the class A image group and the class B image group according to the second prediction mask map.

[0101] Finally, the fusion feature X4 containing the two sets of image information is sent to the decoder to predict the collaborative salient target, and the mask output second prediction mask image is obtained. and the second prediction mask map Specifically, in this embodiment, the white area in the second prediction mask image may be used as the co-salient object in the A-type image group and the B-type image group.

[0102] The process described in the above embodiment is explained below through an embodiment in actual application.

[0103] In order to solve the problem in the prior art that the model is limited by the quality of extracting single image features and common features, resulting in poor image co-salient target detection and segmentation capabilities, this embodiment provides a dual-branch co-salient detection method based on Brownian distance. The method uses Transformer to extract features of a single image, uses a BDC (Brownian distance covariance) module to calculate the similarity within and between image groups using a joint distribution, uses category information for supervision, and extracts more discriminative features, so that the model can more accurately detect and segment co-salient targets between image groups.

[0104] To achieve the above purpose, the present embodiment provides the following technical solution: a dual-branch collaborative saliency detection method based on Brownian distance, based on each image group, each image group corresponds to a target category, and the pictures in each image group contain target objects with different posture features and appearance features: perform the following steps A to G to obtain a dual-branch collaborative saliency detection model, which consists of a T2T (Tokens to Token) structure, a Transformer structure, and a BDC module structure, and then apply the dual-branch collaborative saliency detection model to obtain the segmentation mask of each image.

[0105] See also Figure 2 , Figure 2 A schematic diagram of the workflow of a dual-branch collaborative saliency detection model provided in an embodiment of the present application. The figure shows an image group (Group) and Image patch and T2T structure, class token cls (class token), Transformer structure (Transformer Layer), up sampling (up sample), multi-layer perceptron (MLP), classifier (Classifier), cross entropy loss l ce And the predicted mask map (Output) In the picture Represents the concatenation function concat, Indicates inserting a category element inserttoken. Indicates the removal of the category primitive split token. express The sequence of graphics elements after adding the category graphics element, express The sequence of primitives after adding the category primitives.

[0106] The dual-branch collaborative saliency detection model includes:

[0107] The structures of the first branch and the second branch are the same, which are input layer, embedding layer, T2T structure, Transformer layer, auxiliary classifier, BDC module, upsampling, and MLP layer. The network inputs the image from the input layer, divides the image into multiple blocks in the embedding layer and forms a primitive sequence, uses two T2T structures to fuse and downsample the image, and then passes through the Transformer layer, MLP layer, and Transformer layer in sequence to obtain deep features. The BDC module is used to obtain the inter-group similarity and intra-group similarity of the deep features, and the BDC loss is calculated using the obtained similarity. The auxiliary classifier is used for classification, and the cross entropy loss is calculated using the classification results. Then, bilinear interpolation is used to upsample the deep features that have passed through the Transformer layer, and then passes through the MLP layer. After the first branch and the second branch: the results of the first branch and the second branch are concatenated along the channel dimension, passed through a Transformer layer, and finally predicted using the MLP layer, outputting the mask map of the two groups of images, and then using the mask map and the label to calculate the cross entropy loss and IoU (Intersection over Union, overlap) loss, adding the cross entropy loss, BDC loss, and IoU loss to get the total loss, and using back propagation and SGD stochastic gradient descent algorithm to obtain and update the model parameters. The BDC module and auxiliary classifier are only in the training stage.

[0108] See also Figure 3 , Figure 3 A schematic diagram of a T2T structure provided in an embodiment of the present application is shown in the figure. The T2T structure used in the present embodiment reduces the length of the primitive sequence and expands the channels of the primitive sequence. It consists of two parts: reconstruction and splitting. i , X i ′、X i ' +1 There are three primitive sequences respectively. The T2T structure consists of two stages: reconstruction and splitting. In the reconstruction stage, the primitive sequence is reconstructed into a feature map after passing through a Transformer layer, and in the splitting stage, the feature map is split into primitive sequences again. Figure 3 The T2T structure (process) includes: a sequence of primitives X i , X i ′、X i ' +1 , Transformer structure Transformer Layer, Reshape (change) function. The implementation process includes the reconstruction stage Re-structurization and the splitting stage soft split, I represents the feature map.

[0109] See also Figure 4 , Figure 4 This is a schematic diagram of a Transformer structure provided in an embodiment of the present application. The Transformer structure is mainly composed of multi-head self-attention and MixFFN. The dimension of the primitive sequence remains unchanged after passing through the Transformer structure. Figure 4 The Transformer structure Transformer Layer includes an input sequence Input sequence, a Norm layer, a multi-head self-attention layer Multi-head Self-Attention, a multi-layer perceptron MLP, a convolutional layer Conv and an output sequence Output sequence.

[0110] This embodiment provides a dual-branch collaborative saliency detection method based on Brownian distance, based on two image groups, each image group corresponds to a target category, and the collaborative saliency foreground of the pictures in each image group belongs to the category corresponding to the group, and the following steps A to G are performed to obtain a dual-branch collaborative saliency detection model, and then the dual-branch collaborative saliency detection model is applied to obtain the collaborative saliency detection result of each picture. The dual-branch collaborative saliency detection method based on Brownian distance implemented based on the above dual-branch collaborative saliency detection model may include the following steps:

[0111] Step A: Input two image groups into the two branches of the network respectively. and Recorded as Where N is the number of images in each group, C is the number of channels, H and W are the height and width of each image respectively. When input, C=3, H=W=224. The network first uses an overlapping sliding window to slide on each image to divide each image into multiple image blocks. Here, a convolution kernel with a window size of k=7, stride=4, and padding=2 is used to implement it, dividing each image into 3136 image blocks to form a primitive sequence. Where dim is the number of channels of each image block.

[0112] Step B: Sequence the primitives As input, after two T2T structures, we get the primitive sequence The T2T structure consists of two parts: reconstruction and splitting. In the reconstruction stage, the input primitive sequence X i First, through a Transformer structure, we get the primitive sequence X i '. The Transformer structure in this embodiment consists of two parts: multi-head self-attention and hybrid feedforward network, which are calculated according to the following formula:

[0113] MultiHead(Q,K,V)=Concat(head1,...,head h )W O ;

[0114] Q=T i W q ,K=T i W k ,V=T i W v ;

[0115]

[0116] x out =MixFFN(x in )=MLP(ReLU(Conv(MLP(x in ))));

[0117] Among them, head i For each head of the multi-head self-attention, W O , W q , W k , W v , W i Q , W i K and W i V is a learnable parameter, d k K i The channel dimension, x in and x out are the input and output of the hybrid feedforward network, T i Represents the sequence of image blocks into which the image is divided. i Indicates query, K i Indicates key, V i Indicates value. K i The transpose of .

[0118] In this step, we can use multi-head self-attention to input the primitive sequence X i The information of each element is exchanged, and then the output is added to the input to get x in , and then x in It is sent to the hybrid feedforward network, which consists of two MLPs and a convolution. The role of the convolution is to make up for the lack of position induction bias in the ordinary Transformer structure. out After input x in Add together to get the output X′ of the Transformer structure i, then for X′ i The shape is adjusted to obtain the feature map In the splitting stage, a sliding window with overlap is used to divide the feature map I into multiple image blocks to form a new primitive sequence X′ i+1 , where the primitives in each image block are spliced ​​along the channel dimension. After passing through two T2T structures, the length of the primitive sequence is reduced, the amount of calculation is reduced, and local information is integrated.

[0119] Step C: Two primitive sequences obtained through T2T Two category primitives are inserted into the to supervise the features, learn the category information, and obtain more discriminative features.

[0120] Step D: Take the primitive sequence obtained in step C as input, use a series of Transformer layers to perform feature fusion and deep feature extraction, and fuse global information at the same time to obtain deep fusion features.

[0121] Step E: Separate the deep fusion features obtained in step D into the category primitives to obtain the primitive sequence The category graph is fed into the classifier to get the category prediction, and then the cross entropy loss l is used ce To conduct supervision:

[0122]

[0123] in, is the predicted probability, y is the true label, and y j is the true label of the corresponding category, is the predicted probability of the corresponding category. Through category supervision, the network can learn more discriminative features and better represent the image. i is the loss for each image, N is the number of images, M is the number of categories, and c is the current category.

[0124] The cross entropy loss is calculated using the category prediction results. After the category prediction results are obtained, the cross entropy loss is calculated using the results. The purpose of this operation is to add category information so that the network can learn more discriminative features. This scheme only uses the cross entropy loss for supervision after the category prediction results. The cross entropy loss is calculated using the category prediction results to supervise the category primitives. When the prediction is wrong, the loss is greater, and the more correct the prediction, the smaller the loss.

[0125] At the same time, the primitive sequence can be The image sequences are sent to the BDC module, which is modeled by joint distribution to calculate the inter-group similarity s1 and intra-group similarities s2 and s3 of the image groups. As two random variables X and Y, the BDC is calculated as:

[0126]

[0127] Among them, ||·|| represents the Euclidean norm, Γ represents the complete gamma function, for The channel dimension samples c values ​​with independent and identical distribution {(x1,y1),...,(x c ,y c )}.

[0128] ρ is the joint probability density function established using the BDC matrix, It means that the dimension of the vector is p, and each element is a natural number. is a natural number with dimension q, p and q are the vector dimensions, and t is the range of X The value within the range of s is Y. The value inside, φ X and φ Y are the marginal distribution functions of X and Y respectively.

[0129] φ XY The calculation method is:

[0130]

[0131] Where i is the imaginary unit, t T is the transpose of t, s T is the transpose of s, The channel dimension samples c values, each value is a pixel, consisting of two coordinates, x and y. k and k are the horizontal and vertical coordinates of the kth sampling value respectively.

[0132] For discrete values, BDC has a closed form calculation method, defined as:

[0133]

[0134]

[0135] in, They are x k With x l and k With y l The Euclidean distance between k and k is the horizontal and vertical coordinates of the kth sampling value, x l and l is the horizontal and vertical coordinates of the lth sampling value

[0136] Therefore, the BDC calculation method can be simplified as follows:

[0137] ρ(A,B)=tr(A T B);

[0138]

[0139]

[0140] Where tr(·) is the trace of the matrix, A and B are BDC matrices, A T represents the transposed matrix of matrix A, a kl Denotes the BDC matrix A, b kl Represents the BDC matrix B. Since the BDC matrices A and B are symmetric matrices, then:

[0141]

[0142] Where a is the matrix obtained by taking the upper triangular matrix of A and then vectorizing it, and b is the matrix obtained by taking the upper triangular matrix of B and then vectorizing it. T represents the transposed matrix of a. Finally, this embodiment defines the inter-group similarity s1 and intra-group similarities s2 and s3 of the image group as:

[0143]

[0144]

[0145]

[0146] Then through BDC loss l bdc Supervision is performed to make the distance between groups as large as possible and the distance within groups as small as possible. The formula is as follows:

[0147]

[0148] At the same time After upsampling and MLP operation, the features are obtained And then insert the category element.

[0149] Step F: In the feature After inserting the category primitives, they are spliced ​​along the spatial dimension to obtain a new primitive sequence, which is then further fused through the Transformer structure. After that, feature X4 is obtained by removing the category primitives.

[0150] Step G: Send the fusion feature X4 containing the information of the two groups of images to the decoder, and then divide the obtained results into two groups to obtain the predicted mask map for the two groups of images Among them, the decoder consists of two MLP layers.

[0151] See also Figure 5 , Figure 5 A schematic diagram of the result of a dual-branch collaborative saliency detection based on Brownian distance provided in an embodiment of the present application, from Figure 5 As can be seen from the figure, this embodiment can better segment the co-salient objects.

[0152] This embodiment changes the model to a pure Transformer architecture, changing the previous structure based on convolutional neural networks, so that the model can obtain better features. At the same time, this embodiment uses the BDC matrix to model the similarities within and between groups to obtain more discriminative features. In addition, this embodiment adds a convolutional network in the FFN stage, replacing the original position primitives, and obtains better results.

[0153] The above embodiment discloses a dual-branch collaborative saliency detection method based on Brownian distance. In two branches with shared weights, two image groups of different categories are input respectively, and the network divides each image into multiple image blocks to form two primitive sequences. This embodiment uses the T2T structure to reduce the dimension of the primitive sequence and fuse local information. After that, this embodiment inserts category primitives into the primitive sequence for category supervision and learns more discriminative features. This embodiment uses a series of Transformer layers for feature fusion to obtain deep features. In the Transformer structure, this embodiment removes the position primitives and adds a convolutional network in the FFN stage to add position induction bias. This embodiment uses the BDC module to calculate the inter-group similarity and intra-group similarity of two groups of images using joint distribution to reduce the intra-group distance and increase the inter-group distance. After upsampling, MLP operation, inserting category primitives and splicing the two deep features, this embodiment uses a series of Transformer layers to interact information, and finally removes the category primitives from the interactive features and sends them to the decoder to predict the collaborative saliency map. The collaborative saliency detection algorithm of the present invention still has high accuracy and robustness under the conditions of complex background and diverse interference targets.

[0154] The embodiment of the present application also provides a collaborative salient object detection system, which can be applied to a device running a dual-branch collaborative salient detection model, wherein the dual-branch collaborative salient detection model includes a Tokens-to-Token structure, a Transformer structure, and a Brownian distance covariance calculation module, and the collaborative salient object detection system includes:

[0155] An image division module is used to divide the images in the first image group and the second image group into image blocks to obtain a first image element sequence and the second primitive sequence

[0156] A dimension reduction module is used to reduce the first primitive sequence using a Tokens-to-Token structure. and the second primitive sequence Perform dimensionality reduction and fuse local information to obtain the first primitive sequence and the second primitive sequence

[0157] A fusion module is used to combine the first primitive sequence and the second primitive sequence Insert the corresponding category primitives into each, and use the Transformer structure to transform the first primitive sequence and the second primitive sequence Perform feature fusion operations, deep feature extraction operations and global information fusion operations to obtain deep-level fusion features;

[0158] The primitive processing module is used to remove the category primitives in the deep fusion features to obtain a first primitive sequence and the second primitive sequence

[0159] A parameter adjustment module, used to calculate the first primitive sequence using the Brownian distance covariance calculation module and the second primitive sequence similarity information, and calculating the similarity loss according to the similarity information; and also used to convert the first primitive sequence and the second primitive sequence Inputting the auxiliary classifier to obtain a category prediction result, and calculating a first cross entropy loss according to the category prediction result; and also used to adjust the model parameters of the dual-branch collaborative saliency detection model using the similarity loss and the first cross entropy loss;

[0160] A task receiving module, used to receive a collaborative salient object detection task and determine a class A image group and a class B image group corresponding to the collaborative salient object detection task;

[0161] The task processing module is used to detect the co-salient objects in the A-type image group and the B-type image group using the dual-branch co-saliency detection model.

[0162] This embodiment divides the images in the first image group and the second image group into image blocks to obtain the first primitive sequence and the second primitive sequence, and uses the Tokens-to-Token structure to perform dimensionality reduction processing and local feature fusion on the first primitive sequence and the second primitive sequence. This embodiment also uses the Transformer structure to perform feature fusion operations, deep feature extraction operations and global information fusion operations on the first primitive sequence and the second primitive sequence, and uses the Brownian distance covariance calculation module to calculate the similarity within and between image groups and determine the similarity loss based on the calculated similarity information. This embodiment also uses an auxiliary classifier to calculate the first cross entropy loss, and uses the similarity loss and the first cross entropy loss to adjust the model parameters of the dual-branch collaborative saliency detection model, and then uses the dual-branch collaborative saliency detection model after adjusting the parameters to detect collaborative saliency objects. Therefore, this embodiment can improve the detection accuracy of collaborative saliency objects.

[0163] Furthermore, the parameter adjustment module is further used to adjust the first primitive sequence and the second primitive sequence Perform upsampling and MLP operations to obtain the first primitive sequence and the second primitive sequence Also used in the first primitive sequence and the second primitive sequence Insert the corresponding category of graphics elements into the first graphics element sequence and the second primitive sequence Obtain a first spliced ​​primitive sequence, use a Transformer structure to interact and fuse the first spliced ​​primitive sequence, remove the category primitives, and obtain a fused primitive sequence Also used for using a decoder to decode the fused primitive sequence Performing co-saliency target prediction to obtain a first predicted mask map, calculating a second cross entropy loss and an IoU loss between the first predicted mask map and the corresponding label; and calculating a total loss according to the similarity loss, the first cross entropy loss, the second cross entropy loss and the IoU loss, and using the total loss to adjust the model parameters of the dual-branch co-saliency detection model.

[0164] Furthermore, the process of the task processing module using the dual-branch collaborative saliency detection model to detect the collaboratively salient objects in the A-type image group and the B-type image group includes: dividing the images in the A-type image group and the B-type image group into image blocks to obtain the A-type image element sequence and B-type primitive sequence The A-type primitive sequence is processed by using the Tokens-to-Token structure. and the B-type primitive sequence Perform dimensionality reduction and fuse local information to obtain a sequence of type A graphics elements and B-type primitive sequence In the A-type primitive sequence and the B-type primitive sequence The corresponding category primitives are inserted into the A-category primitives, and the Transformer structure is used to transform the A-category primitive sequence and the B-type primitive sequence Perform feature fusion operation, deep feature extraction operation and global information fusion operation to obtain deep-level fusion features; remove the category primitives in the deep-level fusion features to obtain a class A primitive sequence and B-type primitive sequence For the A-type primitive sequence and B-type primitive sequence Perform upsampling and MLP operations to obtain a sequence of type A graph elements. and B-type primitive sequence In the A-type primitive sequence and the B-type primitive sequence Insert the corresponding category of graphics elements into the A-type graphics element sequence and the B-type primitive sequence A second spliced ​​primitive sequence is obtained, and the second spliced ​​primitive sequence is interacted and fused by using a Transformer structure to remove the category primitives to obtain a fused primitive sequence X4; a decoder is used to perform co-salient target prediction on the fused primitive sequence X4 to obtain a second prediction mask map, and the co-salient objects in the class A image group and the class B image group are determined according to the second prediction mask map.

[0165] Furthermore, the task processing module splices the A-type graphic element sequence and the B-type primitive sequence The process of obtaining the second spliced ​​primitive sequence includes: converting the A-type primitive sequence and the B-type primitive sequence The stitching is performed along the spatial dimension to obtain the second stitching primitive sequence.

[0166] Furthermore, the process of the task processing module determining the co-salient objects in the class A image group and the class B image group according to the second prediction mask map includes: taking the white area in the second prediction mask map as the co-salient objects in the class A image group and the class B image group.

[0167] Furthermore, the process of the image division module dividing the images in the first image group and the second image group into image blocks includes: dividing the images in the first image group and the second image group into image blocks using a sliding window with overlap.

[0168] Furthermore, the dimension reduction module uses the Tokens-to-Token structure to reduce the first primitive sequence and the second primitive sequence The process of performing dimensionality reduction processing and fusing local information includes: converting the first primitive sequence and the second primitive sequence The input is to the two Tokens-to-Token structures connected in series to perform dimensionality reduction and fuse local information.

[0169] Furthermore, the parameter adjustment module uses the Brownian distance covariance calculation module to calculate the first primitive sequence and the second primitive sequence The process of obtaining similarity information includes: using the Brown distance covariance calculation module to calculate the first primitive sequence Corresponding intra-group similarity information; using the Brown distance covariance calculation module to calculate the second primitive sequence Corresponding intra-group similarity information; using the Brown distance covariance calculation module to calculate the first primitive sequence With the second primitive sequence The inter-group similarity information.

[0170] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, which will not be repeated here.

[0171] The present application also provides a storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0172] The present application also provides an electronic device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the electronic device may also include various network interfaces, power supplies and other components.

[0173] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0174] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A collaborative salient object detection method, characterized in that: A device for running a dual-branch collaborative saliency detection model, wherein the dual-branch collaborative saliency detection model includes a Tokens-to-Token structure, a Transformer structure, an auxiliary classifier, and a Brownian distance covariance calculation module, and the collaborative saliency object detection method includes: Divide the images in the first image group and the second image group into image blocks to obtain a first image element sequence and the second primitive sequence The first primitive sequence is coded using the Tokens-to-Token structure. and the second primitive sequence Perform dimensionality reduction and fuse local information to obtain the first primitive sequence and the second primitive sequence In the first primitive sequence and the second primitive sequence Insert the corresponding category primitives into each, and use the Transformer structure to transform the first primitive sequence and the second primitive sequence Perform feature fusion operations, deep feature extraction operations and global information fusion operations to obtain deep-level fusion features; Remove the category primitives in the deep fusion feature to obtain the first primitive sequence and the second primitive sequence The first primitive sequence is calculated using the Brownian distance covariance calculation module and the second primitive sequence similarity information, and calculating a similarity loss according to the similarity information; The first primitive sequence and the second primitive sequence Input the auxiliary classifier to obtain a category prediction result, and calculate a first cross entropy loss according to the category prediction result; Adjusting model parameters of the dual-branch collaborative saliency detection model using the similarity loss and the first cross entropy loss; Receiving a collaborative salient object detection task, and determining a class A image group and a class B image group corresponding to the collaborative salient object detection task; The dual-branch co-saliency detection model is used to detect co-saliency objects in the A-type image group and the B-type image group.

2. The collaborative salient object detection method according to claim 1, characterized in that: After removing the category primitives in the deep fusion feature, a first primitive sequence is obtained. and the second primitive sequence After that, it also includes: For the first primitive sequence and the second primitive sequence Perform upsampling and MLP operations to obtain the first primitive sequence and the second primitive sequence In the first primitive sequence and the second primitive sequence Insert the corresponding category of graphics elements into the first graphics element sequence and the second primitive sequence Obtain a first spliced ​​primitive sequence, use a Transformer structure to interact and fuse the first spliced ​​primitive sequence, remove the category primitives, and obtain a fused primitive sequence The fused primitive sequence is decoded by a decoder Performing collaborative salient target prediction to obtain a first predicted mask map, and calculating a second cross entropy loss and an IoU loss between the first predicted mask map and the corresponding label; Accordingly, adjusting the model parameters of the dual-branch collaborative saliency detection model by using the similarity loss and the first cross entropy loss includes: A total loss is calculated according to the similarity loss, the first cross entropy loss, the second cross entropy loss and the IoU loss, and a model parameter of the dual-branch collaborative saliency detection model is adjusted using the total loss.

3. The collaborative salient object detection method according to claim 1, characterized in that: Detecting the co-salient objects in the A-type image group and the B-type image group using the dual-branch co-saliency detection model includes: The images in the A-type image group and the B-type image group are divided into image blocks to obtain a sequence of A-type image elements. and B-type primitive sequence The A-type primitive sequence is processed by using the Tokens-to-Token structure. and the B-type primitive sequence Perform dimensionality reduction and fuse local information to obtain a sequence of type A graphics elements and B-type primitive sequence In the A-type primitive sequence and the B-type primitive sequence The corresponding category primitives are inserted into the A-category primitive sequence, and the Transformer structure is used to transform the A-category primitive sequence and the B-type primitive sequence Perform feature fusion operations, deep feature extraction operations and global information fusion operations to obtain deep-level fusion features; Remove the category primitives in the deep fusion feature to obtain a class A primitive sequence and B-type primitive sequence For the A-type primitive sequence and B-type primitive sequence Perform upsampling and MLP operations to obtain a sequence of type A graph elements. and B-type primitive sequence In the A-type primitive sequence and the B-type primitive sequence Insert the corresponding category element into the A-category image sequence and the B-type primitive sequence Obtain a second spliced ​​primitive sequence, interact and fuse the second spliced ​​primitive sequence using a Transformer structure, remove the category primitives, and obtain a fused primitive sequence X4; A decoder is used to perform co-salient target prediction on the fused primitive sequence X4 to obtain a second prediction mask map, and co-salient objects in the class A image group and the class B image group are determined according to the second prediction mask map.

4. The collaborative salient object detection method according to claim 3, characterized in that: Splicing the A-type primitive sequence and the B-type primitive sequence The second stitching primitive sequence is obtained, including: The A-type primitive sequence and the B-type primitive sequence The stitching is performed along the spatial dimension to obtain the second stitching primitive sequence.

5. The collaborative salient object detection method according to claim 3, characterized in that: Determining the co-salient objects in the A-type image group and the B-type image group according to the second prediction mask map includes: The white area in the second prediction mask image is used as the co-salient object in the A-type image group and the B-type image group.

6. The collaborative salient object detection method according to claim 1, characterized in that: The dividing the images in the first image group and the second image group into image blocks comprises: The images in the first image group and the second image group are divided into image blocks using sliding windows with overlap.

7. The collaborative salient object detection method according to claim 1, characterized in that: The first primitive sequence is processed by using a Tokens-to-Token structure and the second primitive sequence Perform dimensionality reduction and fuse local information, including: The first primitive sequence and the second primitive sequence The input is to the two Tokens-to-Token structures connected in series to perform dimensionality reduction and fuse local information.

8. The collaborative salient object detection method according to claim 1, characterized in that: The first primitive sequence is calculated using the Brownian distance covariance calculation module and the second primitive sequence Similarity information, including: The first primitive sequence is calculated using the Brownian distance covariance calculation module The corresponding intra-group similarity information; The second primitive sequence is calculated using the Brownian distance covariance calculation module The corresponding intra-group similarity information; The first primitive sequence is calculated using the Brownian distance covariance calculation module With the second primitive sequence The similarity information between groups.

9. A collaborative salient object detection system, characterized in that: A device for running a dual-branch collaborative saliency detection model, wherein the dual-branch collaborative saliency detection model includes a Tokens-to-Token structure, a Transformer structure, an auxiliary classifier, and a Brownian distance covariance calculation module, and the collaborative saliency object detection system includes: An image division module is used to divide the images in the first image group and the second image group into image blocks to obtain a first image element sequence and the second primitive sequence A dimension reduction module is used to reduce the first primitive sequence using a Tokens-to-Token structure. and the second primitive sequence Perform dimensionality reduction and fuse local information to obtain the first primitive sequence and the second primitive sequence A fusion module is used to combine the first primitive sequence and the second primitive sequence Insert the corresponding category primitives into each, and use the Transformer structure to transform the first primitive sequence and the second primitive sequence Perform feature fusion operations, deep feature extraction operations and global information fusion operations to obtain deep-level fusion features; The primitive processing module is used to remove the category primitives in the deep fusion features to obtain a first primitive sequence and the second primitive sequence A parameter adjustment module, used to calculate the first primitive sequence using the Brownian distance covariance calculation module and the second primitive sequence similarity information, and calculating the similarity loss according to the similarity information; and also used to convert the first primitive sequence and the second primitive sequence Inputting the auxiliary classifier to obtain a category prediction result, and calculating a first cross entropy loss according to the category prediction result; and also used to adjust the model parameters of the dual-branch collaborative saliency detection model using the similarity loss and the first cross entropy loss; A task receiving module, used to receive a collaborative salient object detection task and determine a class A image group and a class B image group corresponding to the collaborative salient object detection task; The task processing module is used to detect the co-salient objects in the A-type image group and the B-type image group using the dual-branch co-saliency detection model.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the collaborative salient object detection method according to any one of claims 1 to 8 are implemented.

11. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the steps of the collaborative salient object detection method as described in any one of claims 1 to 8 are implemented.

Images