Object detection method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an object detection method and device, electronic equipment and storage medium, which can be applied to the technical field of computer vision. The method comprises: obtaining features and semantic vectors of each image in a plurality of images by using an encoder; generating common object information based on the features and semantic vectors of the plurality of images by using a consensus extraction module, the common object information being information related to objects having common characteristics in the plurality of images; fusing the common object information and the features of each image by using a consensus distribution module to obtain fused features of the image; and generating a mask map of the common object in the image based on the fused features of each image by using a decoder.

Description

Object detection methods, devices, electronic equipment and storage media Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to an object detection method, apparatus, electronic device and storage medium. Background Technology

[0002] Joint salient object detection is a classic task in computer vision research, aiming to detect salient objects that appear simultaneously in a given set of images. Building upon traditional salient object detection, this task requires not only that the located objects be in the foreground in every image (i.e., salient), but also that objects of that category appear simultaneously in multiple images (i.e., commonality). This task has broad application prospects in object detection, semantic segmentation, image retrieval, and video analysis, and possesses significant research value.

[0003] In the early days, methods for handling this task primarily relied on manually constructed image features. With the rise of deep learning, researchers designed numerous network architectures specifically for this task, significantly improving its performance. However, the detection accuracy of these methods still needs improvement. Summary of the Invention

[0004] Embodiments of this disclosure provide an object detection method, apparatus, electronic device, and storage medium.

[0005] According to one aspect of this disclosure, an object detection method is proposed, comprising: using an encoder to acquire features and semantic vectors of each of multiple images; using a consensus extraction module to generate common object information based on the features and semantic vectors of the multiple images, wherein the common object information is information related to objects with common characteristics in the multiple images; using a consensus distribution module to fuse the common object information with the features of each image to obtain fused features of the image; and using a decoder to generate a mask map of common objects in the image based on the fused features of each image.

[0006] For example, the consensus extraction module includes alternating first and second networks. The process of generating common object information using the consensus extraction module based on the features and semantic vectors of the multiple images includes: for adjacent first and second networks...

[0007] The first network is used to extract common components from the semantic vectors of multiple images to obtain a consensus semantic vector for the multiple images and an updated semantic vector for each image.

[0008] The second network aggregates the features of each image into an updated semantic vector for that image, resulting in a modified semantic vector for that image.

[0009] The consensus semantic vector for the multiple images output by the first network in the last layer serves as common object information.

[0010] For example, the extraction of common components from the semantic vectors of multiple images using the first network includes:

[0011] The first network performs self-attention operations on the semantic vectors of the plurality of images and the mean vector of the semantic vectors of the plurality of images according to the following equation:

[0012]

[0013] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; i = 1, ..., M+1, M+1 represents the number of the first networks in the consensus extraction module, and i represents the first network located at the i-th layer among the M+1 first networks; This represents the self-attention-based operation performed by the first network in the i-th layer; v n，(i) This represents the semantic vector of the nth image input to the first layer of the first network; This represents the consensus semantic vector for the multiple images output by the first network at layer i; This represents the updated semantic vector of the nth image, output by the first network in the i-th layer.

[0014] For example, the step of using a second network to aggregate the features of each image into an updated semantic vector for that image includes:

[0015] The second network performs mutual attention operations on the feature map of each image and the updated semantic vector of that image according to the following equation:

[0016]

[0017] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; i = 1, ..., M, M represents the number of second networks in the consensus extraction module; i represents the second network located at the i-th layer among the M second networks; This represents the feature map of the nth image; This represents the updated semantic vector input to the nth image in the i-th layer of the second network; This represents the mutual attention-based operation performed by the second network in the i-th layer; v n，(i+1) Let represent the modified semantic vector of the nth image output from the second network at layer i; where The query is input into the second network. They are input into the second network as keys and values, respectively.

[0018] For example, the step of using a second network to aggregate the features of each image into an updated semantic vector for that image includes:

[0019] The second network performs self-attention operations on the feature map of each image and the updated semantic vector of that image according to the following equation:

[0020]

[0021] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; i = 1, ..., M, M represents the number of second networks in the consensus extraction module; i represents the second network located at the i-th layer among the M second networks; This represents the feature map of the nth image input to the i-th layer of the second network; This represents the updated semantic vector input to the nth image in the i-th layer of the second network; This represents the self-attention-based operation performed by the second network in the i-th layer; v n，(i+1) This represents the modified semantic vector of the nth image output by the second network in the i-th layer; This represents the modified feature map of the nth image output by the second network model at layer i.

[0022] For example, the consensus distribution module includes a third network and at least one fourth network, and the process of fusing the public object information with the features of each image using the consensus distribution module includes:

[0023] The consensus semantic vector output by the consensus extraction module is applied to the feature map of each image using a third network to obtain a modified consensus semantic vector for that image.

[0024] The consensus semantic vector of each image output by the third network is combined with the feature map of that image to obtain the vector combination of each image;

[0025] The vector combination of multiple images is sequentially passed through the at least one fourth network, wherein each fourth network performs self-attention operation on the consensus semantic vector and feature map of each received image to obtain the updated consensus semantic vector and updated feature map of the image.

[0026] The updated consensus semantic vector of each image output from the fourth network in the last layer is applied to the feature map of that image to obtain the fused features of that image.

[0027] For example, the method further includes: performing aggregation and allocation operations on the updated consensus semantic vectors of multiple images output by at least one fourth network.

[0028] For example, the operation of aggregating and assigning the updated consensus semantic vectors of multiple images output by at least one fourth network includes:

[0029] Calculate the mean vector of the updated consensus semantic vectors of multiple images output by the fourth network;

[0030] The calculated mean vector is concatenated with the updated consensus semantic vector of each image to obtain a concatenated vector for each image.

[0031] The concatenated vectors of multiple images are input into a linear layer to obtain a consensus semantic vector of the multiple images after aggregation and allocation.

[0032] For example, the application of the consensus semantic vector output by the consensus extraction module to the feature map of each image using a third network includes:

[0033] The third network performs mutual attention operations on the consensus semantic vector output by the consensus extraction module and the feature map of each image according to the following equation:

[0034]

[0035] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; Trans CTCA () indicates a mutual attention-based operation performed by the third network layer; Linear CTCA () denotes a linear transformation; g represents the consensus semantic vector output by the consensus extraction module; F L，n This represents the feature map of the nth image; Let represent the modified consensus semantic vector of the nth image; where Linear CTCA (g) is input into the third network as a query, F L，n They are input into the third network as keys and values, respectively.

[0036] For example, the mutual attention-based operations performed by the third network layer involve processing the queries, keys, and values ​​input to the third network according to the following equation:

[0037] Attn_Result=Add&Norm(MHA(Q,K,V)·γ Attn Q)

[0038] Result=Add&Norm(FFN(Attn_Result)·γ FFNAttn_Result)

[0039] Where MHA() represents multi-head attention operation, Add&Norm() represents addition operation with layer normalization, and FFN() represents feedforward network; Q, K, and V are the query, key, and value input to the third network, respectively; γ Attn γ represents the weights for the results of multi-head attention operations. FFN Represents the weights for the feedforward network, where γ Attn and γ FFN All are learnable channel-by-channel weights.

[0040] For example, each fourth network performs self-attention operations on the consensus semantic vector and feature map of each received image according to the following equation:

[0041]

[0042] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; i = 1, ..., M′, M′ represents the number of fourth networks in the consensus distribution module, and i represents the fourth network located at the i-th layer among the M′ fourth networks; F represents the consensus semantic vector of the nth image input to the fourth network of the i-th layer; L，n，(i) This represents the feature map of the nth image input to the fourth network of the i-th layer; This represents the self-attention-based operation performed by the fourth network in the i-th layer; F represents the updated consensus semantic vector from the output of the fourth network at layer i to the nth image at layer i+1; L，n，(i+1) This represents the updated feature map of the nth image output from the fourth network at layer i to the fourth network at layer i+1.

[0043] For example, the updated consensus semantic vectors of multiple images output by at least one fourth network are aggregated and assigned according to the following equation.

[0044]

[0045] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; i = 1, ..., M′-1, M′ represents the number of fourth networks in the consensus distribution module, and i represents the fourth network located at the i-th layer among the M′ fourth networks; This represents the consensus semantic vector of the nth image output by the fourth network in the i-th layer; This represents the linear transformation performed in the aggregation and distribution operations of the fourth network at layer i; This represents the consensus semantic vector after aggregation and distribution.

[0046] For example, applying the updated consensus semantic vector of each image output by the fourth network at the last layer to the feature map of that image includes:

[0047] The updated consensus semantic vector of each image output by the fourth network in the last layer is adjusted in dimension by a linear transformation according to the following equation, and then multiplied element-wise by the feature map of the image output by the intermediate layer of the encoder and the updated feature map of the image output by the fourth network in the last layer:

[0048]

[0049]

[0050] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; M' represents the number of the fourth network in the consensus distribution module; L represents the number of layers in the encoder; l ∈ {l1, ..., l K-1}, where l is the sequence number of the selected layer in the encoder's L layers, and K is an integer greater than 1; F L，n The feature map representing the nth image output from the l-th layer of the encoder; This represents the consensus semantic vector of the nth image output by the fourth network at layer M'; F represents the linear transformation performed for multiplication with the feature map output from the l-th layer of the encoder; L，n，(M′+1) This represents the updated feature map of the nth image output by the fourth network in layer M'; This represents the linear transformation performed to multiply the updated feature map of the nth image output by the fourth network at layer M'.

[0051] For example, the method further includes adding the normalized mean of the semantic vectors of the plurality of images to the public object information before fusing the public object information with the features of each image using the consensus distribution module.

[0052] For example, the parameters of the encoder are initialized based on the pre-trained weights of the contrastive language-image pre-training.

[0053] For example, the method of generating a mask map of common objects in an image using a decoder based on the fusion features of each image includes:

[0054] Generate a mask map of common objects in an image based on the following equation:

[0055]

[0056] Where Conv represents a 1×1 convolutional layer, used to reduce the number of channels to 1 to obtain a mask image; Up k This represents a series of 3×3 convolutions and 2× bilinear interpolations used to gradually increase the spatial dimension; k = 1, ..., K-1; k = K.

[0057] For example, the method further includes: supervising the training of a network model based on binary cross-entropy loss and cross-union ratio loss, wherein the network model includes the encoder, the consensus extraction module, the consensus distribution module, and the decoder.

[0058] According to another aspect of this disclosure, an object detection apparatus is provided, comprising: an encoding unit for acquiring features and semantic vectors of each of a plurality of images using an encoder; an extraction unit for generating common object information based on the features and semantic vectors of the plurality of images using a consensus extraction module, wherein the common object information is information related to objects having common characteristics in the plurality of images; a distribution unit for fusing the common object information with the features of each image using a consensus distribution module to obtain fused features of the image; and a decoding unit for generating a mask image of common objects in the image using a decoder based on the fused features of each image.

[0059] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0060] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0061] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above. Attached Figure Description

[0062] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0063] Figure 1 schematically illustrates an application scenario of the object detection method according to an embodiment of the present disclosure;

[0064] Figure 2 schematically illustrates a flowchart of an object detection method according to an embodiment of the present disclosure;

[0065] Figure 3 schematically illustrates a neural network model used in an object detection method according to an embodiment of the present disclosure;

[0066] Figure 4A schematically illustrates a flowchart of the consensus extraction process according to an embodiment of the present disclosure;

[0067] Figure 4B schematically illustrates a consensus extraction module according to an embodiment of the present disclosure;

[0068] Figure 5A schematically illustrates a flowchart of a consensus distribution process according to an embodiment of the present disclosure;

[0069] Figure 5B schematically illustrates a consensus distribution module according to an embodiment of the present disclosure;

[0070] Figure 6 schematically illustrates the experimental results of the object detection method according to an embodiment of the present disclosure;

[0071] Figure 7 schematically illustrates a block diagram of an object detection apparatus according to an embodiment of the present disclosure; and

[0072] Figure 8 schematically illustrates a structural diagram of an electronic device suitable for implementing an object detection method according to an embodiment of the present disclosure. Detailed Implementation

[0073] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0074] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0075] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0076] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0077] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0078] Joint salient object detection based on deep learning typically involves four stages: encoding, consensus extraction, consensus distribution, and decoding. In consensus extraction, traditional methods primarily obtain consensus by filtering local features or taking a (weighted) average of local features. While this captures the most representative local information of common objects, it struggles to provide a complete, semantic-level representation of the common object category. Therefore, these methods may fail when common objects exhibit significant intra-class differences and may incorrectly detect irrelevant objects that share local similarities with the target object but do not belong to the same category. Furthermore, in consensus distribution, traditional methods mainly achieve consensus distribution through concatenation, addition, element-wise multiplication, dot products, or combinations of these basic operations. However, common objects may exhibit very different attributes, appearances, and poses in different images, and these simple operations cannot reflect this diversity, potentially leading to poor performance in handling difficult examples. Some traditional methods have explicitly or implicitly considered the image-specific differences in consensus. However, they fail to obtain a compact and unified consensus representation, potentially resulting in inconsistent detection targets across different images within the same group and increasing computational costs.

[0079] Embodiments of this disclosure provide an object detection method, comprising: acquiring features and semantic vectors of each of multiple images using an encoder; generating common object information based on the features and semantic vectors of the multiple images using a consensus extraction module, wherein the common object information is information related to objects with common characteristics in the multiple images; fusing the common object information with the features of each image using a consensus distribution module to obtain fused features of the image; and generating a mask map of common objects in the image using a decoder based on the fused features of each image. The object detection method of this disclosure designs two modules for semantic-level consensus extraction and image-specific consensus distribution, achieving object detection with a simple model structure and exhibiting higher detection accuracy compared to existing methods.

[0080] Figure 1 schematically illustrates an application scenario of the object detection method according to an embodiment of the present disclosure.

[0081] As shown in Figure 1, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0082] Users can interact with server 105 via network 104 using at least one of the first terminal device 101, second terminal device 102, and third terminal device 103 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, second terminal device 102, and third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0083] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0084] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0085] It should be noted that the object detection method provided in this embodiment can generally be executed by server 105. Correspondingly, the object detection device provided in this embodiment can generally be located in server 105. The object detection method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the object detection device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0086] It should be understood that the number of terminal devices, networks, and servers shown in Figure 1 is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0087] Figure 2 schematically illustrates a flowchart of an object detection method according to an embodiment of the present disclosure.

[0088] In operation S210, the encoder is used to obtain the feature and semantic vector of each image in multiple images.

[0089] For example, the original visual Transformer model can be used as an image encoder. The Transformer is a neural network architecture for sequence modeling, widely used in natural language processing and computer vision tasks, such as machine translation and the Vision Foundation Model (ViT). The core idea of ​​the Transformer is to use a self-attention mechanism to capture the dependencies between different positions in the input sequence. The self-attention mechanism dynamically calculates the relevance weights of each position in the sequence to its other positions, effectively capturing global contextual information. The Transformer consists of multiple stacked encoders and decoders. The encoder is responsible for encoding the input sequence, while the decoder progressively generates the target sequence based on the encoder's output. In the encoder, the input at each position interacts with information through a self-attention layer, and then undergoes feature transformation through a feedforward neural network layer. This stacked layer structure allows the model to extract and combine feature representations of the input sequence layer by layer. In the decoder, in addition to the self-attention layer and the feedforward neural network layer, another attention mechanism is introduced, called encoder-decoder attention. This attention mechanism allows the decoder to focus on the encoder's output during sequence generation, thus making better use of the information in the encoder. By using the Transformer, the model can learn long-range dependencies in the input sequence and has the advantage of parallel computation, making training and inference faster. Therefore, the Transformer has become an important model in the field of natural language processing, achieving remarkable results on multiple tasks.

[0090] The encoder parameters can be initialized based on the pre-trained weights of Contrastive Language-Image Pretraining (CLIP). CLIP is a multimodal deep learning mechanism that can process images and text simultaneously and map them to a shared feature space. CLIP's design goal is to enable models to understand and represent the semantic relationships between images and text. To achieve this goal, CLIP uses contrastive learning. Contrastive learning is a self-supervised learning method that learns the similarities and differences between samples by comparing positive and negative samples. In CLIP, images and text are encoded separately to obtain corresponding feature representations. Then, by maximizing the similarity of positive samples while minimizing the similarity of negative samples, the model is trained to map related image-text pairs to nearby feature spaces, while mapping unrelated image-text pairs to more distant feature spaces. In this way, CLIP learns shared representations between images and text, enabling the model to interact and understand between images and text. For example, CLIP can perform tasks such as image search, image classification, or text description generation by calculating the similarity between images and text. CLIP's advantage lies in the fact that it does not require any supervised labels; it only needs to be pre-trained from large-scale image and text data through contrastive learning. This gives CLIP broad application potential and demonstrates remarkable performance in multiple tasks and domains.

[0091] In operation S220, the consensus extraction module is used to generate common object information based on the features and semantic vectors of multiple images. The common object information is information related to objects with common characteristics in multiple images.

[0092] For example, the consensus extraction module may include alternating first and second networks. The first network can be used to extract common components from the semantic vectors of multiple images, obtaining a consensus semantic vector for the multiple images and an updated semantic vector for each image. The second network can be used to aggregate the features of each image into the updated semantic vector for that image, obtaining a refined semantic vector for that image. The consensus semantic vector for the multiple images output by the first network at the last layer can be used as common object information.

[0093] By utilizing an alternating first and second network, the consensus extraction module can perform two steps alternately: a global step and a local step. In the global step, the semantic representation of the image is adjusted using the first network, and consensus extraction is performed simultaneously. In the local step, local information of the image is processed, and features of salient objects are aggregated into the corresponding semantic vectors.

[0094] In operation S230, the consensus distribution module is used to fuse the public object information with the features of each image to obtain the fused features of the image.

[0095] For example, the consensus distribution module may include a third network and at least one fourth network. The third network can be used to apply the consensus semantic vector output by the consensus extraction module to the feature map of each image, obtaining a modified consensus semantic vector for that image. Then, the consensus semantic vector of each image output by the third network is combined with the feature map of that image to obtain a vector combination for each image. The vector combinations of multiple images can then be sequentially passed through at least one fourth network. Each fourth network performs self-attention operations on the received consensus semantic vector and feature map of each image to obtain an updated consensus semantic vector and updated feature map for that image. The updated consensus semantic vector of each image output by the fourth network at the last layer can be applied to the feature map of that image to obtain the fused features of that image.

[0096] In some embodiments, aggregation and allocation (A&A) operations can also be performed on the updated consensus semantic vectors of multiple images output by each fourth network.

[0097] In operation S240, the decoder generates a mask map of common objects in the image based on the fusion features of each image.

[0098] For example, a feature pyramid network-like decoder can be used, whose input includes not only the image feature map fused with consensus information, but also the output of the encoder's intermediate layer.

[0099] Figure 3 schematically illustrates a neural network model used in an object detection method according to an embodiment of the present disclosure.

[0100] As shown in Figure 3, the neural network model includes an encoder 310, a consensus extraction module 320, a consensus distribution module 330, and a decoder 340.

[0101] The encoder 310 segments the received input image into non-overlapping blocks. The pixel values ​​within each block are flattened into a vector, and a linear transformation converts these vectors into a sequence of label embeddings. The encoder 310 can be implemented using a visual Transformer to extract the feature and semantic vectors of each image from multiple images. This visual Transformer introduces a learnable category label, which, along with the block labels, is fed into several Transformer layers. Each layer consists of self-attention operations, a feedforward network, and residual connections. The Transformer structure allows the model to efficiently process each image block in parallel and model global dependencies between images.

[0102] In existing methods, joint salient object detection models that typically use Transformer-based encoders often employ some variant of the visual Transformer to better suit dense prediction tasks. In contrast, using the original visual Transformer model decouples the training process from the requirements of downstream tasks. This makes the neural network model of the present disclosure compatible with various research advancements related to the visual Transformer, and allows for the convenient incorporation of the latest pre-training techniques, scalability achievements, etc.

[0103] The CLIP pre-trained visual Transformer weights can be selected as the initial parameters for encoder 310. The CLIP model adds a linear layer to the end of the original visual Transformer, and the category label after linear transformation is regarded as the CLIP image feature vector. At the same time, CLIP also uses another Transformer to encode the sentence matching the image into CLIP text feature vectors. CLIP performs cross-modal contrastive learning on these two feature vectors to train the image and text Transformer encoders. Choosing the CLIP visual Transformer weights has the following two advantages. First, CLIP's large-scale contrastive learning pre-training can endow its image features with rich and high-quality semantic information. Second, CLIP uses sentences describing the image as supervision information. Compared with other supervision signals such as image category, sentences can provide a more complete and comprehensive description of the image semantics. When the content of the image is relatively complex, the image features learned under the guidance of sentence description are less likely to miss key information about joint salient objects.

[0104] For example, given a set of N input images The encoder 310 can encode the input image according to the following equation:

[0105]

[0106]

[0107] Where n = 1, ..., N, l = {2, ..., L} are the layer indices of encoder 310; c 0，n This is the initialization of the category flag; c l,n It is the category flag of the output of the lth layer of encoder 310, F l,n This is the feature map output from the l-th layer of the encoder. For the category label c of the last layer... L,n and feature map F L,n Following the CLIP model's specifications, a linear transformation can be applied to obtain the CLIP vector and CLIP feature map:

[0108]

[0109]

[0110] Where n = 1, ..., N.

[0111] The consensus extraction module 320 generates common object information based on the features and semantic vectors of multiple images output by the encoder 310. Common object information, also known as consensus, is information related to objects with common characteristics in the multiple images. For example, as shown in Figure 3, if multiple input images all contain an accordion, then the accordion is a common object. The consensus extraction module 310 can extract information related to the accordion from the features and semantic vectors of the multiple images output by the encoder 310 as common object information. In some embodiments, the consensus extraction module 310 may include an alternately configured first network and a second network to alternately perform global and local steps. In the global step, the first network is used to adjust the semantic representation of the image and simultaneously perform consensus extraction. In the local step, the second network takes effect and processes each image separately, and so on. The first network, located at the last layer, ultimately outputs the common object information. The first network can be a Group-based Semantic Agreement Transformer (GSAT) layer, hereinafter also referred to as the GSAT layer. The second network can be an Image-based Local Information Aggregation Transformer (ILIAT) layer, also referred to below as the ILIAT layer. The ILIAT layer processes the local information of the image and aggregates the features of salient objects into corresponding semantic vectors; while the GSAT layer realizes semantic-level information exchange, extracting common components from the semantics of each image. Furthermore, the GSAT layer updates the semantic vector of each image, and these vectors can participate in the next ILIAT layer, providing more reliable guidance for the aggregation of local information. This will be described in detail below.

[0112] The consensus distribution module 330 fuses the common object information with the features of each image to obtain the fused features of the image. The consensus distribution module 330 can be a Transformer-based module. For example, the consensus distribution module may include a third network and at least one fourth network. The third network may be a Contextualization Transformer with Cross-Attention (CTCA) layer, hereinafter referred to as the CTCA layer. The fourth network may be an Image-based Transformer for Dispersion (ITD) layer, hereinafter referred to as the ITD layer. The third network places the consensus representation within the specific context of each image. Next, the modified consensus vector is concatenated with the corresponding image feature map and fed into at least one fourth network for processing. Furthermore, to further promote information exchange between images within the group, an aggregation and allocation (A&A) operation is inserted between adjacent ITD layers. This is implemented by calculating the mean of a set of consensus flags output by the ITD, concatenating the mean with each consensus flag, and feeding it into a linear layer. This will be described in detail below.

[0113] Decoder 340 can generate a mask map of common objects in an image based on the fused features of each image. Decoder 340 can be a feature pyramid network-like decoder, whose input can include not only the image features fused with consensus information from the consensus distribution module, but also the outputs of intermediate layers of encoder 310. Unlike convolutional neural networks and some hierarchical variations of the visual Transformer, the shape of the feature maps in the original visual Transformer model remains unchanged throughout, and the outputs of earlier layers do not necessarily contain more fine-grained information than the outputs of later layers. For example, decoder 340 can decode according to the following equation:

[0114]

[0115] Where Conv represents a 1×1 convolutional layer, used to reduce the number of channels to 1 to obtain a mask image; Up k This represents a series of 3×3 convolutions and 2× bilinear interpolations used to progressively increase the spatial dimension; k = 1, ..., K, F k This represents the K inputs to the decoder; Dec(F) 1 F K ) indicates the output of the decoder.

[0116] The consensus extraction process of this disclosure embodiment will now be described with reference to Figures 4A and 4B.

[0117] Figure 4A schematically illustrates a flowchart of the consensus extraction process according to an embodiment of the present disclosure.

[0118] According to embodiments of this disclosure, the consensus extraction module may include an alternately configured first network and a second network, and the consensus extraction process may include the following operations S2201 to S2205.

[0119] In operation S2201, the first layer first network is used to extract common components from the semantic vectors of multiple input images to obtain consensus semantic vectors for the multiple images and updated semantic vectors for each image.

[0120] In operation S2202, the features of each image are aggregated into the updated semantic vector for that image using the first layer second network, resulting in the modified semantic vector of that image.

[0121] The second network can perform processing based on mutual attention or self-attention operations.

[0122] This completes one round of alternating processing between the first and second networks. Then, similar operations are performed using the second-layer first network and the second-layer second network, and so on.

[0123] In operation S2203, the first network of layer M extracts common components from the semantic vectors of multiple input images to obtain a consensus semantic vector for the multiple images and an updated semantic vector for each image. This step can be implemented in a similar manner to S2201 above, and will not be described in detail here.

[0124] In operation S2204, the features of each image are aggregated into the updated semantic vector for that image using the Mth layer of the second network (i.e., the last layer of the second network), resulting in the modified semantic vector for that image. This step can be implemented in a similar manner to S2202 above, and will not be described in detail here.

[0125] In operation S2205, the common components are extracted from the semantic vectors of multiple input images using the first network of the (M+1)th layer (that is, the last first network layer), to obtain the consensus semantic vector for the multiple images and the updated semantic vector for each image.

[0126] Figure 4B schematically illustrates a consensus extraction module according to an embodiment of the present disclosure.

[0127] As shown in Figure 4B, the consensus extraction module includes an M+1 layer first network 3201_1, 3201_2, ..., 3201_M+1 and an M layer second network 3202_1, 3202_2, ..., 3201_M, where M is an integer greater than or equal to 1. The first and second networks are alternated. Each first network can include a Transformer. Each second network can include one or more Transformers. Transformers in the second network can share parameters.

[0128] The consensus extraction module is based on the CLIP feature map provided by the encoder. and semantic vectors The model alternates between two steps: in the global step, the GSAT layer adjusts the semantic representation of the image and simultaneously extracts consensus; in the local step, the ILIAT layer takes effect and processes each image separately. This will be explained in detail below.

[0129] As shown in Figure 4B, the encoder extracts the features (also called CLIP block markers) and semantic vectors (also called CLIP category markers) of images 1, 2, 3, and 4 respectively, and provides them to the consensus extraction module. In the consensus extraction module, the first network (also called the GSAT layer) 3201_1 located in the first layer can extract common components from the semantic vectors of the multiple input images (i.e., images 1, 2, 3, and 4) provided by the encoder in the manner described above, to obtain the consensus semantic vector for the multiple images and the updated semantic vector for each image.

[0130] For example, the first network can perform self-attention operations on the semantic vectors of the plurality of images and the mean vector of the semantic vectors of the plurality of images according to the following equation:

[0131]

[0132] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; i = 1, ..., M+1, M+1 represents the number of the first networks in the consensus extraction module, and i represents the first network located at the i-th layer among the M+1 first networks; This represents the self-attention-based operation performed by the first network in the i-th layer; v n,(i) This represents the semantic vector of the nth image input to the first layer of the first network; This represents the consensus semantic vector for the multiple images output by the first network at layer i; This represents the updated semantic vector of the nth image, output by the first network in the i-th layer. The initial v... n，(1) It can be

[0133] Then, as shown in Figure 4B, taking image 1 as an example, the second network (also called the ILIAT layer) 3202_1 in layer 1 aggregates the features of image 1 (also called CLIP block markers) output by the encoder into the updated semantic vector for image 1 output by the first network 3201_1 in layer 1 (i.e., the CLIP category markers output by the first network 3201_1), thus obtaining the modified semantic vector of image 1 (i.e., the CLIP category markers output by the second network 3202_1 for image 1). The processing methods for images 2, 3, and 4 are similar and will not be described in detail here.

[0134] As an example, the second network can perform mutual attention operations on the feature map of each image and the updated semantic vector of that image according to the following equation:

[0135]

[0136] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; i = 1, ..., M, M represents the number of second networks in the consensus extraction module; i represents the second network located at the i-th layer among the M second networks; This represents the feature map of the nth image; This represents the updated semantic vector input to the nth image in the i-th layer of the second network; This represents the mutual attention-based operation performed by the second network in the i-th layer; v n，(i+1) Let represent the modified semantic vector of the nth image output from the second network at layer i; where The query is input into the second network. They are input into the second network as keys and values, respectively. Initially... It can be

[0137] As another example, the second network can perform self-attention operations on the feature map of each image and the updated semantic vector of that image according to the following equation:

[0138]

[0139] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; i = 1, ..., M, M represents the number of second networks in the consensus extraction module; i represents the second network located at the i-th layer among the M second networks; This represents the feature map of the nth image input to the i-th layer of the second network; This represents the updated semantic vector input to the nth image in the i-th layer of the second network; This represents the self-attention-based operation performed by the second network in the i-th layer; v n，(i+1) This represents the modified semantic vector of the nth image output by the second network in the i-th layer; This represents the modified feature map of the nth image output by the second network model at layer i. As shown in Figure 4B, when the second network uses self-attention operation, in addition to outputting semantic vectors (i.e., CLIP category labels), the second network also outputs image features (i.e., CLIP block labels).

[0140] This completes one round of alternating processing between the first and second networks.

[0141] Then, similar operations are performed using the second layer first network and the second layer second network. For example, the second layer first network 3201_2 can extract common components from the semantic vectors provided by the previous layer (i.e., the first layer) second network 3202_1 to obtain a consensus semantic vector and an updated semantic vector for each image. Then, the second layer second network 3202_2 aggregates the features of each image output by the encoder into the updated semantic vector for that image output by the second layer first network 3201_2, to obtain the refined semantic vector of the image.

[0142] Finally, as shown in Figure 4B, the first network 3201_M+1 of the M+1 layer extracts the consensus semantic vector from the semantic vector output by the second network 3202_M of the M layer, and uses it as the public object information.

[0143] In some embodiments, after the first network 3201_M+1 outputs common information at layer M+1, as shown in Figure 4B, the semantic vectors of multiple images output by the encoder can be normalized and averaged to obtain a normalized mean, which is then added to the common object information. Finally, the common object information with the added normalized mean is provided to the consensus distribution module.

[0144] For example, the mean of the CLIP vectors of each image output by the encoder, after L2 normalization, can be added to the consensus representation according to the following equation:

[0145]

[0146] Norm() represents normalization. This represents the consensus semantic vector output by the first network 3201_M+1 at layer M+1. This represents the CLIP vector of the nth image output by the encoder.

[0147] The consensus extraction module of this embodiment employs alternating GSAT and ILIAT layers. The entire consensus extraction process consists of M+1 layers of GSAT and M layers of ILIAT, with the output of the average semantic tag corresponding to the last GSAT layer being regarded as the final consensus representation. Intuitively, the ILIAT layer processes local information of the image and aggregates the features of salient objects into the corresponding semantic vectors; while the GSAT layer realizes semantic-level information exchange, extracting common components from the semantics of each image. In addition, the GSAT layer updates the semantic vector of each image, and these vectors can participate in the next ILIAT layer, providing more reliable guidance for the aggregation of local information. Furthermore, by adding the mean of the CLIP vectors of each image output by the encoder after L2 normalization to the consensus representation, the important semantic priors given to the CLIP vectors during the pre-training process can be fully utilized.

[0148] The consensus distribution process of an embodiment of this disclosure will now be described with reference to Figures 5A and 5B.

[0149] Figure 5A schematically illustrates a flowchart of a consensus distribution process according to an embodiment of the present disclosure.

[0150] According to embodiments of this disclosure, the consensus distribution module may include a third network and at least one fourth network, and the consensus distribution process may include the following operations S2301 to S2304.

[0151] In operation S2301, the consensus semantic vector output by the consensus extraction module is applied to the feature map of each image using the third network to obtain a modified consensus semantic vector for that image.

[0152] In operation S2302, the consensus semantic vector of each image output by the third network is combined with the feature map of that image to obtain a vector combination for each image.

[0153] In operation S2303, the vector combination of multiple images is sequentially passed through at least one fourth network, wherein each fourth network performs self-attention operation on the consensus semantic vector and feature map of each received image to obtain the updated consensus semantic vector and updated feature map of that image.

[0154] In operation S2304, the updated consensus semantic vector of each image output from the fourth network in the last layer is applied to the feature map of that image to obtain the fused features of that image.

[0155] For example, the updated consensus semantic vector of each image output by the fourth network in the last layer can be adjusted in dimension by a linear transformation, and then multiplied element-wise by the feature map of the image output by the intermediate layer of the encoder and the updated feature map of the image output by the fourth network in the last layer, respectively.

[0156] In some embodiments, the updated consensus semantic vectors of multiple images output by the fourth network can also be aggregated and distributed. For example, the mean vector of the updated consensus semantic vectors of multiple images output by the fourth network can be calculated; the calculated mean vector can be concatenated with the updated consensus semantic vector of each image to obtain a concatenated vector for each image; the concatenated vectors of multiple images can be input into a linear layer to obtain the aggregated and distributed consensus semantic vectors of the multiple images.

[0157] Figure 5B schematically illustrates a consensus distribution module according to an embodiment of the present disclosure.

[0158] As shown in Figure 5B, the consensus distribution module may include a third network 3301 and M' fourth networks 3302_1, 3302_2, ..., 3302_M', where M' is an integer greater than or equal to 1. The third network 3301 may be a contextualized Transformer layer with mutual attention, hereinafter referred to as CTCA. The input to the third network 3301 includes queries, keys, and values. Using the Transformer-based third network 3301 for consensus distribution is beneficial for handling the differences in public objects across different images. Each of the fourth networks 3302_1, 3302_2, ..., 3302_M' may be an image-based distribution Transformer layer, hereinafter referred to as an ITD layer.

[0159] First, the third network 3301 applies the consensus semantic vector (i.e., consensus flag) output by the consensus extraction module to the feature map of each image (images 1 to 4) output by the encoder (each image's feature map includes multiple feature vectors, i.e., block flags), to obtain a modified consensus semantic vector (i.e., modified consensus flag) for each image. In this way, the consensus representation is placed in the specific context of each image.

[0160] For example, the third network 3301 can perform mutual attention operations on the consensus semantic vector output by the consensus extraction module and the feature map of each image according to the following equation:

[0161]

[0162] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; Trans CTCA () indicates a mutual attention-based operation performed by the third network layer; Linear CTCA () denotes a linear transformation used to adjust the number of channels; g represents the consensus semantic vector output by the consensus extraction module; F L,nThis represents the feature map of the nth image; Let represent the modified consensus semantic vector of the nth image; where Linear CTCA (g) is input into the third network as a query, F L,n These are input as keys and values, respectively, to the third network. In this way, the third network 3301 outputs... Represents common objects in image I n The specific states and attributes within.

[0163] In some embodiments, unlike a traditional Transformer, the third network 3301 can multiply the results of mutual attention and the results of the feedforward network by a learnable channel-wise weight before adding them to the residual connections. Therefore, the Transformer in the third network 3301 is denoted as Trans-L. As an example, the mutual attention-based operations performed by the third network layer include processing the query, key, and value input to the third network according to the following equation:

[0164] Attn_Result=Add&Norm(MHA(Q,K,V)·γ Attn Q)

[0165] Result=Add&Norm(FFN(Attn_Result)·γ FFN (9)

[0166] Where MHA() represents multi-head attention operation, Add&Norm() represents addition operation with layer normalization, and FFN() represents feedforward network; Q, K, and V are the query, key, and value input to the third network, respectively; γ Attn γ represents the weights for the results of multi-head attention operations. FFN Represents the weights for the feedforward network, where γ Attn and γ FFN All are learnable channel-by-channel weights.

[0167] Compared to the traditional Transformer structure, this design of the third network structure in the embodiments of this disclosure enables the network to adaptively adjust the degree of contextualization.

[0168] Next, the consensus semantic vector of each image output by the third network 3301 is combined with the feature map of that image to obtain a vector combination for each image. For example, the modified consensus vector of image 1 output by the third network 3301 is concatenated with the feature map of image 1 to obtain the vector combination of image 1; the modified consensus vector of image 2 output by the third network 3301 is concatenated with the feature map of image 2 to obtain the vector combination of image 2, and so on.

[0169] Then, the vectors from multiple images are sequentially passed through the fourth network 3302_1, 3302_2, ..., 3302_M'. Each fourth network performs self-attention operations on the consensus semantic vector and feature map of each received image according to the following equation:

[0170]

[0171] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; i = 1, ..., M′, M′ represents the number of fourth networks in the consensus distribution module, and i represents the fourth network located at the i-th layer among the M′ fourth networks; F represents the consensus semantic vector of the nth image input to the fourth network of the i-th layer; L，n，(i) This represents the feature map of the nth image input to the fourth network of the i-th layer; This represents the self-attention-based operation performed by the fourth network in the i-th layer; F represents the updated consensus semantic vector from the output of the fourth network at layer i to the nth image at layer i+1; L，n，(i+1) This represents the updated feature map of the nth image output from the fourth network at layer i to the fourth network at layer i+1.

[0172] In some embodiments, to further facilitate information exchange among images within a group, aggregation and allocation operations can be performed on the output of at least one fourth network. For example, aggregation and allocation operations can be performed on the output of each fourth network except the last fourth network layer, thereby inserting aggregation and allocation (A&A) operations between adjacent ITD layers. The aggregation and allocation operation may include calculating the mean of a set of consensus flags output by the ITD layer, concatenating the mean with each consensus flag, and feeding it into a linear layer. For example, aggregation and allocation operations can be performed on the updated consensus semantic vectors of multiple images output by the fourth network according to the following equation:

[0173]

[0174] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; i = 1, ..., M′-1, M′ represents the number of fourth networks in the consensus distribution module, and i represents the fourth network located at the i-th layer among the M′ fourth networks; This represents the consensus semantic vector of the nth image output by the fourth network in the i-th layer; This represents the linear transformation performed in the aggregation and distribution operations of the fourth network at layer i; This represents the consensus semantic vector after aggregation and distribution.

[0175] In the example shown in Figure 5B, the modified consensus flag output by the third network 3301, combined with the feature map of the corresponding image, is fed into the fourth network 3302_1 located in the first layer. The first-layer fourth network 3302_1 performs self-attention operations on the consensus semantic vector and feature map in each vector combination, thereby generating updated feature maps and semantic vectors for each image. The semantic vector output by the first-layer fourth network 3302_1 is aggregated and assigned (A&A), and then fed as input to the second-layer fourth network 3302_2 along with the feature map output by the first-layer fourth network 3302_1. The second-layer fourth network 3302_2 performs self-attention operations on the updated feature map output by the first-layer fourth network 3302_1 and the A&A-processed semantic vector in a similar manner, and the result is fed to the third-layer fourth network 3302_3, and so on. Finally, the M'-th layer fourth network 3302_M' outputs the updated consensus semantic vector and updated feature map for each image.

[0176] Through this exchange, simple samples (where common objects can be easily located) can help more difficult samples in the same group (where common objects are more concealed or subject to strong interference) to better focus on the target object. Furthermore, the ITD module, which introduces A&A operations, provides greater flexibility to the distribution process (achieved through multiple layers of Transformers) while ensuring that the distribution process does not deviate significantly from the previously extracted consensus.

[0177] Finally, the updated consensus semantic vector of each image output by the fourth network 3302_M' of layer M' can be applied to the feature map of the image to obtain the fused features of the image.

[0178] For example, the updated consensus semantic vector of each image output by the fourth network at the last layer is adjusted in dimension by a linear transformation according to the following equation, and then multiplied element-wise by the feature map of the image output by the intermediate layer of the encoder and the updated feature map of the image output by the fourth network at the last layer, respectively:

[0179]

[0180]

[0181] Where n = 1, ..., N, N represents the number of the plurality of images, and n represents the nth image among the N images; M' represents the number of the fourth network in the consensus distribution module; L represents the number of layers in the encoder; l ∈ {l1, ..., l K-1}, where l is the sequence number of the selected layer in the encoder's L layers, and K is an integer greater than 1; F L,n The feature map representing the nth image output from the l-th layer of the encoder; This represents the consensus semantic vector of the nth image output by the fourth network at layer M'; F represents the linear transformation performed for multiplication with the feature map output from the l-th layer of the encoder; L，n，(M′+1) This represents the updated feature map of the nth image output by the fourth network in layer M'; This represents the linear transformation performed to multiply the updated feature map of the nth image output by the fourth network at layer M'. In this way, the basic distribution implemented by element-wise multiplication is preserved; that is, the modified consensus vector output by the last ITD layer is adjusted in dimension by a linear transformation and then multiplied element-wise onto the feature map of the corresponding image and the output of the intermediate layers of the encoder.

[0182] The consensus distribution module generates K outputs and This can be provided to the decoder for decoding, thereby obtaining the joint saliency mask prediction. For example, the decoder can perform decoding according to equation (3) as described above:

[0183]

[0184] Where Conv represents a 1×1 convolutional layer, used to reduce the number of channels to 1 to obtain a mask image; Up k This represents a series of 3×3 convolutions and 2× bilinear interpolations used to progressively increase the spatial dimension; k = 1, ..., K, F k This represents the K inputs to the decoder; Dec(F) 1 F K ) represents the decoder output. Where, k = 1, ..., K-1; k = K.

[0185] In the above embodiments, the first network, the second network, the third network and the fourth network are all implemented based on Transformer. The self-attention / mutual attention-based operations performed by each Transformer include self / mutual attention calculation, feedforward network, residual connection, etc., which will not be described in detail here.

[0186] The objective of the joint salient object detection task can be formally represented as: given a set of images containing objects of the same class, and where each object is located in a salient region. Predict a set of joint saliency masks These masks need to accurately cover jointly salient objects in the corresponding images. First, all input images are resized to the same size, thus… Correspondingly, there is M n ∈[0,1] 1×H×W .

[0187] In some embodiments, the neural network model used in the methods of this disclosure can be trained in an end-to-end manner. For example, the network model can be trained under supervised supervision based on binary cross-entropy loss and cross-union ratio loss. As an example, given mask prediction... and manually labeled mask The training objective is to minimize the binary cross-entropy loss:

[0188]

[0189] Combined loss and intersection:

[0190]

[0191] The sum of the two is the total loss function:

[0192] L total =L BCE +L IoU (15)

[0193] The superior performance of the method of this disclosure embodiment on the joint salient object detection task will be illustrated below with reference to the experimental results in Tables 1 to 3 and Figure 6.

[0194] We evaluated the proposed model on three commonly used datasets in this field: Cosal2015, CoSOD3k, and CoCA. Cosal2015 has relatively simple samples, while CoSOD3k and CoCA are two challenging datasets because each image group in these datasets contains many interfering objects, and common objects often exhibit significant morphological differences across different images. Evaluation metrics included the maximum E-index, the S-index, the maximum F-index, and the mean absolute error (N / A).

[0195] Currently, there are three commonly used supervised training sets in the field of joint salient object detection: COCO-9213, DUTS_class, and COCO-SEG. In our experiments, we trained our model on a combination of COCO-SEG and DUTS_class, consistent with the previously best-performing method. In each column of Tables 1 to 3, bold indicates the best result, and underline indicates the second-best result.

[0196] Table 1 shows a comparison of the performance of the method of this disclosure embodiment on the CoCA dataset and the experimental results of existing methods.

[0197]

[0198] Table 1

[0199] Table 2 shows a comparison of the performance of the method of this disclosure embodiment on the CoSOD3k dataset with the experimental results of existing methods.

[0200]

[0201] Table 2

[0202] Table 3 shows a comparison of the performance of the method of this disclosure embodiment on the CoSal2015 dataset with the experimental results of existing methods.

[0203]

[0204] Table 3

[0205] As shown in Tables 1 to 3, the traditional CoSFormer, GCoNet+, and MCCL achieve optimal performance on the CoSal2015, CoCA, and CoSODk datasets, respectively. The method of this disclosure performs comparably to GCoNet+ on CoCA, while its process and objective function design are significantly simpler than those of GCoNet+ and MCCL. The method of this disclosure performs particularly well on CoSOD3k, significantly outperforming all existing methods (e.g., a +1.9% increase in the maximum E-metric and a +2.0% increase in the maximum F-metric). The method of this disclosure also outperforms CoSFormer on the vast majority of metrics. It can be seen that the method of this disclosure achieves overall superior quantitative performance compared to existing methods.

[0206] Figure 6 schematically illustrates the experimental results of the object detection method according to embodiments of the present disclosure. Figure 6 shows a visualization of the prediction results of the method of the present disclosure and existing methods. Figure 6 contains four types of examples, including the normal case and three typical hard cases. "Small objects" require the model to not miss minute details in the image. By utilizing the rich semantic information in the CLIP feature vector and effectively aggregating local information, the method of the present disclosure can detect objects at very small scales. "Interference" indicates the presence of significant interfering objects in the image, while "difference" indicates that the appearance of common objects varies greatly in different images. Methods relying solely on local information may be misled by such complex scenarios, thus detecting some incorrect objects or ignoring certain parts of common objects. In contrast, the method of the present disclosure avoids these problems through semantic-level consensus extraction and image-specific consensus distribution, enabling it to efficiently obtain complete information about common objects and perform accurate detection.

[0207] The embodiments of this disclosure design two novel modules for semantic-level consensus extraction and image-specific consensus distribution. For consensus extraction, a hierarchical Transformer model is designed. This model alternately performs two steps: first, it aggregates the salient object information of each image into the image's category label; second, it aggregates the common information from a set of category labels into a consensus label. For consensus distribution, an image-specific consensus distribution module is designed. This module first uses an attention mechanism to adjust the consensus, adding image-specific common object details. Then, it uses multiple Transformer layers to implement the distribution process, introducing intra-group interactions between these Transformer layers, allowing simpler examples to help more difficult examples achieve better distribution results. Furthermore, the embodiments of this disclosure utilize a pre-trained visual Transformer as an encoder to provide high-quality image features; and a decoder similar to a feature pyramid network is used to predict a joint saliency mask.

[0208] Furthermore, traditional joint salient object detection methods often introduce additional tasks or model branches to improve performance. These include utilizing additional image classification tasks, leveraging prior information from salient object detection tasks, employing different contrastive learning methods, and using adversarial training. While these techniques improve the final model performance, they also lead to model complexity and introduce a large number of hyperparameters to be tuned. Additionally, some methods require additional supervision during the training phase, such as class labels and saliency maps of individual images.

[0209] In this regard, the network model used in the method of this disclosure embodiment has no additional design except for the mask generation process. Experiments show that this simple model structure based on high-quality consensus extraction and distribution can achieve performance comparable to or even better than previous models containing complex modules.

[0210] According to embodiments of this disclosure, a hierarchical consensus extraction module is designed. This module utilizes the high-level semantic information contained in the category labels of the Transformer to obtain a complete representation of a common object category, thereby avoiding the ambiguity and incompleteness problems caused by using local information as consensus. The module also includes an iterative improvement process that gradually improves the quality of image semantic information and group consensus information.

[0211] According to embodiments of this disclosure, a novel consensus distribution module is proposed, capable of distributing consensus information to image features in an image-specific manner, thereby taking into account the diversity of common objects across different images. This module also introduces intra-group interaction during the distribution process, thereby maintaining a balance between commonality and specificity.

[0212] According to embodiments of this disclosure, the two novel modules described above are combined with a visual Transformer encoder and a feature pyramid decoder to build a joint salient object detection model that can be trained end-to-end. This model does not involve any additional modules or auxiliary loss functions beyond joint saliency mask prediction, and its structure is simple and clear. We validate the effectiveness of this model through experimental results on several commonly used datasets.

[0213] Figure 7 schematically shows a block diagram of an object detection apparatus according to an embodiment of the present disclosure.

[0214] As shown in FIG7, the object detection device 700 of this embodiment includes an encoding unit 710, an extraction unit 720, a distribution unit 730 and a decoding unit 740.

[0215] The encoding unit 710 is used to acquire the feature and semantic vector of each of the multiple images using an encoder. In one embodiment, the encoding unit 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0216] Extraction unit 720 is used to generate common object information based on the features and semantic vectors of multiple images using the consensus extraction module. The common object information is information related to objects with common characteristics in the multiple images. In one embodiment, extraction unit 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0217] The distribution unit 730 is used to fuse the public object information with the features of each image using the consensus distribution module to obtain the fused features of the image. In one embodiment, the distribution unit 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0218] Decoding unit 740 is configured to generate a mask map of common objects in an image using the fusion features of each image from the decoder. In one embodiment, decoding unit 740 may be configured to perform the operation S240 described above, which will not be repeated here.

[0219] Figure 8 schematically illustrates a block diagram of an electronic device suitable for implementing an object detection method according to an embodiment of the present disclosure.

[0220] As shown in FIG8, an electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0221] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0222] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0223] Embodiments of this disclosure also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0224] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0225] Embodiments of this disclosure also provide a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the object detection method provided in the embodiments of this disclosure.

[0226] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0227] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0228] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0229] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0230] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0231] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0232] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An object detection method, comprising: The encoder is used to obtain the feature and semantic vector of each image in multiple images; The consensus extraction module generates common object information based on the features and semantic vectors of the multiple images, including: for adjacent first and second networks, the first network extracts common components from the semantic vectors of the multiple images to obtain a consensus semantic vector for the multiple images and an updated semantic vector for each image. The updated semantic vector for each image is obtained by performing a self-attention operation on the semantic vectors of the multiple images and their mean vector. The second network aggregates the features of each image into the updated semantic vector for that image to obtain a modified semantic vector for that image. The consensus extraction module includes alternating first and second networks. The consensus semantic vector for the multiple images output by the last layer of the first network serves as the common object information. The current layer of the second network provides its semantic vector to the next layer of the first network. The common object information relates to objects with common characteristics in the multiple images. The consensus distribution module fuses the common object information with the features of each image to obtain a fused feature for that image. The decoder generates a mask image of the common objects in the image based on the fused feature of each image.

2. The method according to claim 1, wherein, The step of extracting common components from the semantic vectors of multiple images using a first network includes: the first network performing self-attention operations on the semantic vectors of the multiple images and the mean vector of the semantic vectors of the multiple images according to the following equation: in, N represents the number of the plurality of images, and n represents the nth image among the N images; M+1 represents the number of the first network in the consensus extraction module. This represents the first network located in the i-th layer among M+1 first networks; This represents the self-attention-based operation performed by the first network in the i-th layer; This represents the semantic vector of the nth image input to the first layer of the first network; This represents the consensus semantic vector for the multiple images output by the first network at layer i; This represents the updated semantic vector of the nth image, output by the first network in the i-th layer.

3. The method according to claim 1, wherein, The step of aggregating the features of each image into the updated semantic vector for that image using the second network includes: the second network performing mutual attention operations on the feature map of each image and the updated semantic vector of that image according to the following equation: in, N represents the number of the plurality of images, and n represents the nth image among the N images; M represents the number of second networks in the consensus extraction module; This represents the second network located in the i-th layer among M second networks; This represents the feature map of the nth image; This represents the updated semantic vector input to the nth image in the i-th layer of the second network; This represents the mutual attention-based operation performed by the second network in the i-th layer; Let represent the modified semantic vector of the nth image output from the second network at layer i; where The query is input into the second network. They are input into the second network as keys and values, respectively.

4. The method according to claim 1, wherein, The step of aggregating the features of each image into the updated semantic vector for that image using the second network includes: the second network performing self-attention operations on the feature map of each image and the updated semantic vector of that image according to the following equation: in, N represents the number of the plurality of images, and n represents the nth image among the N images; M represents the number of second networks in the consensus extraction module; This represents the second network located in the i-th layer among M second networks; This represents the feature map of the nth image input to the i-th layer of the second network; This represents the updated semantic vector of the nth image input to the i-th layer of the second network; This represents the self-attention-based operation performed by the second network in the i-th layer; This represents the modified semantic vector of the nth image output by the second network in the i-th layer; This represents the modified feature map of the nth image output by the second network model at layer i.

5. The method according to claim 1, wherein, The consensus distribution module includes a third network and at least one fourth network. The process of fusing the public object information with the features of each image using the consensus distribution module includes: applying the consensus semantic vector output by the consensus extraction module to the feature map of each image using the third network to obtain a modified consensus semantic vector for that image; combining the consensus semantic vector of each image output by the third network with the feature map of that image to obtain a vector combination for each image; sequentially passing the vector combinations of multiple images through the at least one fourth network, wherein each fourth network performs self-attention operations on the received consensus semantic vector and feature map of each image to obtain an updated consensus semantic vector and updated feature map for that image; and applying the updated consensus semantic vector of each image output by the fourth network at the last layer to the feature map of that image to obtain the fused features of that image.

6. The method according to claim 5, further comprising: Aggregation and allocation operations are performed on the updated consensus semantic vectors of multiple images output by at least one fourth network.

7. The method according to claim 6, wherein, The aggregation and allocation operation of the updated consensus semantic vectors of multiple images output by at least one fourth network includes: calculating the mean vector of the updated consensus semantic vectors of multiple images output by the fourth network; concatenating the calculated mean vector with the updated consensus semantic vector of each image to obtain a concatenated vector for each image; and inputting the concatenated vectors of multiple images into a linear layer to obtain the aggregated and allocated consensus semantic vectors of the multiple images.

8. The method according to claim 5, wherein, The step of applying the consensus semantic vector output by the consensus extraction module to the feature map of each image using a third network includes: the third network performing mutual attention operations on the consensus semantic vector output by the consensus extraction module and the feature map of each image according to the following equation: in, N represents the number of the plurality of images, and n represents the nth image among the N images; () indicates a mutual attention-based operation performed by the third network layer; Represents a linear transformation; This represents the consensus semantic vector output by the consensus extraction module; This represents the feature map of the nth image; Let represent the modified consensus semantic vector of the nth image; where The query is input into the third network. They are input into the third network as keys and values, respectively.

9. The method according to claim 8, wherein, The mutual attention-based operations performed by the third network layer involve processing the queries, keys, and values ​​input to the third network according to the following equation: Where MHA() represents multi-head attention operation, FFN() represents a summation operation with layer normalization, and FFN() represents a feedforward network. 、 、 These are the query, key, and value input to the third network, respectively. This represents the weights for the results of multi-head attention operations. This represents the weights for the feedforward network, where and All are learnable channel-by-channel weights.

10. The method according to claim 5, wherein, Each fourth network performs self-attention operations on the consensus semantic vector and feature map of each received image according to the following equation: in, N represents the number of the plurality of images, and n represents the nth image among the N images; M' represents the number of fourth networks in the consensus distribution module, and i represents the fourth network located at the i-th layer among the M' fourth networks; This represents the consensus semantic vector of the nth image input to the fourth network of the i-th layer; This represents the feature map of the nth image input to the fourth network of the i-th layer; This represents the self-attention-based operation performed by the fourth network in the i-th layer; This represents the updated consensus semantic vector from the output of the fourth network at layer i to the nth image at layer i+1; This represents the updated feature map of the nth image output from the fourth network at layer i to the fourth network at layer i+1.

11. The method according to claim 6, wherein, The updated consensus semantic vectors of multiple images output by at least one fourth network are aggregated and assigned according to the following equation: in, N represents the number of the plurality of images, and n represents the nth image among the N images; M' represents the number of fourth networks in the consensus distribution module, and i represents the fourth network located at the i-th layer among the M' fourth networks; This represents the consensus semantic vector of the nth image output by the fourth network in the i-th layer; This represents the linear transformation performed in the aggregation and distribution operations of the fourth network at layer i; This represents the consensus semantic vector after aggregation and distribution.

12. The method according to claim 5, wherein, The step of applying the updated consensus semantic vector of each image output by the fourth network at the last layer to the feature map of that image includes: adjusting the dimension of the updated consensus semantic vector of each image output by the fourth network at the last layer through a linear transformation according to the following equation, and multiplying it element-wise by the feature map of the image output by the intermediate layer of the encoder and the updated feature map of the image output by the fourth network at the last layer, respectively: in, N represents the number of the plurality of images, n represents the nth image among the N images; M' represents the number of the fourth network in the consensus distribution module; L represents the number of layers in the encoder; l is the sequence number of the selected layer in the L layers of the encoder, and K is an integer greater than 1; The feature map representing the nth image output from the l-th layer of the encoder; This represents the consensus semantic vector of the nth image output by the fourth network at layer M'; This represents the linear transformation performed for multiplication with the feature map output from the l-th layer of the encoder; This represents the updated feature map of the nth image output by the fourth network in layer M'; This represents the linear transformation performed to multiply the updated feature map of the nth image output by the fourth network at layer M'.

13. The method according to any one of claims 1 to 12, further comprising: Before fusing the common object information with the features of each image using the consensus distribution module, the normalized mean of the semantic vectors of the multiple images is added to the common object information.

14. The method according to any one of claims 1 to 12, wherein, The encoder parameters are initialized based on the pre-trained weights of the contrastive language-image pre-training.

15. The method according to claim 12, wherein, The step of generating a mask map of common objects in an image using a decoder based on the fusion features of each image includes: generating a mask map of common objects in the image according to the following equation: Where Conv represents a 1×1 convolutional layer, used to reduce the number of channels to 1 to obtain a mask image; This represents a series of 3×3 convolutions and 2× bilinear interpolations used to gradually increase the spatial dimension; ， ； ， 。 16. The method according to any one of claims 1 to 12, further comprising: The network model is trained under supervision using binary cross-entropy loss and cross-union ratio loss. The network model includes the encoder, the consensus extraction module, the consensus distribution module, and the decoder.

17. An object detection device, comprising: The encoding unit is used to obtain the feature and semantic vector of each image in multiple images using the encoder; The extraction unit, used by the consensus extraction module to generate common object information based on the features and semantic vectors of the multiple images, includes: for adjacent first and second networks, using the first network to extract common components from the semantic vectors of the multiple images, obtaining a consensus semantic vector for the multiple images and an updated semantic vector for each image, wherein the updated semantic vector for each image is obtained by performing a self-attention operation on the semantic vectors of the multiple images and the mean vector of the semantic vectors of the multiple images; and using the second network to aggregate the features of each image into the updated semantic vector for that image, obtaining a modified semantic vector for that image. The consensus extraction module includes an alternately configured first network and a second network. The consensus semantic vector output by the first network at the last layer for the multiple images serves as common object information. The second network at the current layer provides the semantic vector output by the second network at the current layer to the first network at the next layer. The common object information is information related to objects with common characteristics in the multiple images. The distribution unit is used to fuse the common object information with the features of each image using the consensus distribution module to obtain the fused features of the image. The decoding unit is used to generate a mask map of common objects in the image using the decoder based on the fused features of each image.

18. An electronic device comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-16.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-16.

20. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-16.

Images