Collaborative significance detection method, product, equipment and storage medium

Abstract

The invention discloses a collaborative saliency detection method, a product, equipment and a storage medium, and relates to the field of computer vision, and the method comprises the steps: obtaining at least two current to-be-detected target images, and obtaining a current image group; inputting the current image group into the trained collaborative saliency detection model to extract target text information related to collaborative saliency among the target images in the current image group based on a pre-training language model, and based on the target text information, guiding a segmentation cutting model to segment the collaborative saliency target and background in the current image group to obtain a current collaborative saliency detection result. According to the invention, the capability of identifying the collaborative saliency target and the interference background can be improved, and the accuracy of collaborative saliency detection is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a collaborative saliency detection method, product, device and storage medium. Background Art

[0002] At present, since Co-Saliency Object Detection (CoSOD) technology can help people better extract the main information from a large amount of image or video data, it has been widely used in tasks such as image segmentation, image retrieval, and image quality assessment. However, the semantic category information required for co-salient objects is unknown. This information depends on the specific content of a given image group and needs to be inferred by a designed algorithm. Therefore, how to effectively distinguish co-salient objects from interfering backgrounds in the absence of high-quality semantics is a major challenge facing current co-salient detection technology.

[0003] With the rapid development of deep learning, deep learning technology with powerful image representation extraction capabilities has been widely used in co-saliency detection tasks and has made significant progress. The current mainstream co-saliency detection methods include single-group co-saliency detection methods and multi-group co-saliency detection methods. Among them, the single-group co-saliency detection method is to learn discriminative representations by inputting a group of images, mining intra-group relationships, and extracting binary masks for this group of images to distinguish co-salient targets from the background; while the multi-group co-saliency detection method specifically inputs two or more groups of images, and on the basis of mining intra-group relationships, introduces inter-group differences, and then learns discriminative representations through intra-group-inter-group relationships, and thereby provides binary masks for multiple groups of images respectively.

[0004] However, both of the above co-saliency detection methods use semantically irrelevant binary masks as supervisory information, lack the guidance of semantic labels, and find it difficult to obtain high-quality semantic information, which results in the limitations of the learned discriminative representations and makes it impossible for the model to effectively distinguish co-salient targets from interfering backgrounds within the image. Summary of the invention

[0005] In view of this, the purpose of this application is to provide a collaborative saliency detection method, product, device and storage medium, which can improve the recognition ability of collaborative saliency targets and interference backgrounds, and improve the accuracy of collaborative saliency detection. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses a collaborative saliency detection method, comprising:

[0007] Acquire at least two target images to be detected to obtain a current image group;

[0008] Input the current image group into the trained co-saliency detection model; the co-saliency detection model is a model obtained by training a pre-trained language model and a segmentation model based on an image dataset containing co-saliency objects;

[0009] A current co-saliency detection result corresponding to the current image group output by the co-saliency detection model is obtained; wherein the current co-saliency detection result is a result obtained by extracting target text information related to the co-saliency between target images in the current image group based on a pre-trained language model, and guiding the segmentation model to segment the co-salient targets and background in the current image group based on the target text information. The current co-saliency detection result includes a mask result generated based on the target text information.

[0010] Optionally, the collaborative saliency detection method further includes:

[0011] Acquire an image dataset containing co-salient objects, and group images in the image dataset according to different co-salient objects to obtain multiple co-salient image groups;

[0012] Inputting the images in each co-salient image group into the pre-trained language model in turn to extract features of each image in the co-salient image group to obtain image encoding features, and extracting features of the preset interactive text to obtain text encoding features;

[0013] Decoding the image encoding features and text encoding features corresponding to each image in the co-salient image group to obtain the scene structure knowledge description text corresponding to each image containing the relationship between different co-salient objects;

[0014] Encode each scene structure knowledge description text into text features respectively, and extract keywords from the text features to obtain prompt keyword features corresponding to each scene structure knowledge description text;

[0015] Mining the connections between different prompt keyword features to generate co-salient text prompt words;

[0016] The co-salient text prompt words are input into the segmentation model to convert the co-salient text prompt words into binary masks, thereby obtaining a co-salient detection model including historical co-salient detection results.

[0017] Optionally, feature extraction is performed on each image in the collaboratively salient image group to obtain image coding features, and feature extraction is performed on a preset interactive text to obtain text coding features, including:

[0018] Using an image feature extractor to extract features from each image in the co-salient image group to obtain image coding features, and converting the image coding features into vector representation to obtain an image feature vector;

[0019] Using a text feature extractor to extract features from a preset interactive text to obtain text encoding features, and converting the text encoding features into vector representations to obtain text feature vectors;

[0020] Among them, the image feature extractor and the text feature extractor are located in the pre-trained language model.

[0021] Optionally, decoding the image encoding features and text encoding features corresponding to each image in the co-salient image group to obtain the scene structure knowledge description text corresponding to each image containing the relationship between different co-salient objects, including:

[0022] Combining the image feature vector and the text feature vector corresponding to each image in the co-salient image group to obtain a combined feature vector;

[0023] The combined feature vector is input into the first decoder of the pre-trained language model to decode the combined feature vector to obtain the scene structure knowledge description text corresponding to each image containing the relationship between different co-saliency objects.

[0024] Optionally, each scene structure knowledge description text is encoded into a text feature, and keywords in the text feature are extracted to obtain prompt keyword features corresponding to each scene structure knowledge description text, including:

[0025] Use the text encoder to encode the text of each scene structure knowledge description to obtain text features, and perform linear mapping on the text features to obtain mapped features;

[0026] The multi-head self-attention mechanism is used to enhance the mapped features to obtain enhanced features;

[0027] The keyword extractor is used to extract the keywords in the enhanced features to obtain multiple prompt keyword features corresponding to the knowledge description text of each scene structure.

[0028] Optionally, the connections between different prompt keyword features are mined to generate co-salient text prompt words, including:

[0029] Use the collaborative attention mechanism to mine the connections between different prompt keyword features and obtain collaborative text prompt features;

[0030] The collaborative text cue features are decoded to obtain collaborative salient text cue words.

[0031] Optionally, the collaborative attention mechanism is used to mine the connections between different prompt keyword features to obtain collaborative text prompt features, including:

[0032] Combining multiple prompt keyword features corresponding to each scene structure knowledge description text to obtain a keyword group;

[0033] Input the keyword groups into different linear layers respectively, and multiply the output results of different linear layers to obtain the attention affinity graph of the keyword group;

[0034] Extract the first number of values ​​in the attention affinity graph row by row, and calculate the sum of the first number of values ​​extracted to obtain a first sum;

[0035] Calculating the ratio of the first sum value to the first quantity to obtain a keyword weight, and multiplying the keyword weight by the prompt keyword feature in the keyword group element by element to obtain a plurality of calculation results;

[0036] Extracting first second number of values ​​from the plurality of calculation results, and calculating the sum of the extracted second number of values ​​to obtain a second sum;

[0037] The ratio of the second sum value to the second quantity is calculated to obtain the collaborative text prompt feature.

[0038] Optionally, during the training of the pre-trained language model and the segmentation model, the following steps are also included:

[0039] The real co-salient category text is used as the supervision signal to supervise the learning process of the keyword extractor and the co-attention mechanism.

[0040] Optionally, during the training of the pre-trained language model and the segmentation model, the following steps are also included:

[0041] Obtain co-salient text labels from the image dataset, and perform high-dimensional mapping on the co-salient text labels to obtain true co-prompt features;

[0042] The distance between the true collaborative prompt feature and the collaborative text prompt feature is taken as the loss and the loss is supervised.

[0043] Optionally, the co-salient text prompt words are converted into binary masks to obtain a co-salient detection model including historical co-salient detection results, including:

[0044] Perform feature encoding on the co-salient text prompt words to obtain text feature encoding;

[0045] Performing feature coding on the images in each collaboratively salient image group to obtain image feature coding;

[0046] The image feature encoding and the corresponding text feature encoding are decoded to convert the co-salient text prompt words into binary masks to obtain the historical co-saliency detection results, so as to complete the training of the pre-trained language model and the segmentation model and obtain the co-saliency detection model.

[0047] Optionally, feature encoding is performed on the co-salient text prompt words to obtain text feature encoding, including:

[0048] Inputting the co-salient text prompt words into a text prompt encoder to perform feature encoding on the co-salient text prompt words to obtain text feature encoding;

[0049] Accordingly, feature encoding is performed on the images in each collaboratively salient image group to obtain image feature encoding, including:

[0050] The images in each co-salient image group are sequentially input into an image encoder to perform feature encoding on each image in the co-salient image group to obtain image feature encoding;

[0051] Among them, the text hint encoder and image encoder are located in the segmentation model.

[0052] Optionally, the image feature encoding and the corresponding text feature encoding are decoded to convert the co-salient text prompt words into binary masks to obtain historical co-salient detection results, including:

[0053] Respectively represent the image feature coding and the corresponding text feature coding by vectors to obtain the text prompt vector and the image coding vector;

[0054] Combine the text prompt vector and the corresponding image encoding vector to obtain a combined vector;

[0055] The combined post-vector is input into the second decoder located in the segmentation model to decode the combined post-vector to obtain a historical co-saliency detection result including a co-saliency mask.

[0056] In a second aspect, the present application discloses a computer program product, including a computer program / instruction, which implements the aforementioned collaborative saliency detection method when executed by a processor.

[0057] In a third aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein the processor implements the aforementioned collaborative saliency detection method when executing a computer program stored in the memory.

[0058] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned collaborative saliency detection method is implemented.

[0059] It can be seen that the present application first obtains at least two target images to be detected to obtain the current image group; the current image group is input into the trained co-saliency detection model; the co-saliency detection model is a model obtained by training a pre-trained language model and a segmentation model based on an image data set containing co-saliency objects; the current co-saliency detection result corresponding to the current image group output by the co-saliency detection model is obtained; wherein the current co-saliency detection result is the result obtained by extracting target text information related to the co-saliency between target images in the current image group based on the pre-trained language model, and guiding the segmentation model based on the target text information to segment the co-saliency targets and background in the current image group, and the current co-saliency detection result includes a mask result generated based on the target text information. The present application pre-trains a pre-trained language model and a segmentation model based on an image data set containing co-salient objects to obtain a model for co-saliency detection. When co-saliency detection is required, the current image group to be detected is input into the model, so that the pre-trained language model is used to extract the target text information related to the co-saliency between the target images in the current image group, and the segmentation model is guided based on the extracted target text information to segment the co-salient targets and backgrounds in the current image group, so as to obtain a co-saliency detection result including a mask result generated based on the target text information; the present application uses a pre-trained language model and a segmentation model to solve the problem of extracting co-saliency information from a cross-modal perspective, because the pre-trained language model can perform multi-modal interaction, establish cross-modal implicit semantic relationships, and And it has the ability to understand image-text, so it can generate credible text scene descriptions for images, and the segmentation everything model has excellent instance segmentation effect. Therefore, through the combination of pre-trained language model and segmentation everything model, the difficult problem of image collaborative relationship extraction can be transformed into a simpler text collaborative relationship extraction problem, and high-quality saliency-related text information can improve the ability to distinguish between co-salient targets and interference backgrounds in co-saliency detection. In addition, using high-quality saliency-related text information to guide the segmentation everything model to segment co-salient targets can break through the limitation of semantically irrelevant binary masks as supervisory signals, which helps to alleviate the problem of limited representation learning caused by training scene limitations, thereby further improving the ability to recognize co-salient targets and interference backgrounds, and thus improving the accuracy of co-saliency detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0061] Figure 1 A flowchart of a collaborative saliency detection method disclosed in this application;

[0062] Figure 2 A flowchart of a specific collaborative saliency detection method disclosed in this application;

[0063] Figure 3 A specific collaborative saliency detection model training diagram disclosed in this application;

[0064] Figure 4 A schematic diagram of a specific multi-head self-attention structure disclosed in this application;

[0065] Figure 5 A specific collaborative text prompt feature extraction schematic diagram disclosed in this application;

[0066] Figure 6 A schematic diagram of a specific keyword weight calculation process disclosed in this application;

[0067] Figure 7 A schematic diagram of loss supervision in a specific model training process disclosed in this application;

[0068] Figure 8 A schematic diagram of a collaborative significance detection result disclosed in this application;

[0069] Fig. 9 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0071] The present application discloses a method for detecting collaborative significance. Figure 1 As shown, the method includes:

[0072] Step S11: Acquire at least two target images to be detected to obtain a current image group.

[0073] In this embodiment, when co-saliency detection is required, at least two target images to be subjected to co-saliency detection are first acquired to obtain a current image group.

[0074] Step S12: inputting the current image group into the trained co-saliency detection model; the co-saliency detection model is a model obtained by training a pre-trained language model and a segmentation model based on an image dataset containing co-saliency objects.

[0075] In this embodiment, after the current image group to be detected is obtained, the images in the current image group are further input into the co-saliency detection model obtained by training the pre-trained language model and the segmentation everything model (SAM, Segment Anything Model, a Transformer-based visual model, which can take images and user prompts as inputs and output a segmentation mask to represent the label of each pixel) based on the image data set containing the co-saliency objects, that is, the co-saliency objects in the current image group are detected. Among them, the pre-trained language model can be a multimodal large language model (Multimodal Large Language Model, MLLM), such as mPlug, GPT (Generative Pre-trained Transformer, a language model based on artificial intelligence technology) and other models, which are used to establish cross-modal implicit semantic relationships, have the ability to understand image-text, and can generate credible text scene descriptions for images; the SAM model combined with prompt learning can achieve cross-modal fusion of text and image, and achieve excellent instance segmentation (Instance Segmentation) effect.

[0076] It should be pointed out that the pre-trained language model has powerful implicit semantic knowledge, which is a deep understanding of the relationship between different objects in the image. Using this semantic knowledge in the co-saliency detection task is conducive to breaking through the limitation of semantically irrelevant binary masks as supervisory signals; in addition, the use of the SAM model helps to alleviate the problem of limited representation learning caused by training scene restrictions. In summary, the co-saliency detection model based on the pre-trained language model and the segmentation everything model can learn high-quality semantic knowledge, and high-quality semantic knowledge has an important influence on the discrimination of co-saliency targets and backgrounds. Therefore, by using the SAM model and based on high-quality semantic knowledge to perform instance segmentation of co-saliency targets and interference backgrounds in the image group, the accuracy of co-saliency target and background recognition can be improved, thereby improving the effect of co-saliency detection.

[0077] For details, see Figure 2 As shown, the process of creating a collaborative saliency detection model may include:

[0078] Step S21: acquiring an image dataset containing co-salient objects, and grouping images in the image dataset according to different co-salient objects to obtain a plurality of co-salient image groups;

[0079] In this embodiment, images containing co-salient objects are first collected to obtain an image data set, and then the images in the image data set are grouped according to different co-salient objects to obtain multiple co-salient image groups, where each group contains N images, and each co-salient image group can be expressed as:

[0080] ;

[0081] In the formula, n is a variable, H represents height, W represents width, are images in the co-salient image group.

[0082] Step S22: sequentially inputting images in each co-salient image group into the pre-trained language model to extract features of each image in the co-salient image group to obtain image coding features, and extracting features of the preset interactive text to obtain text coding features;

[0083] In this embodiment, after obtaining multiple collaborative salient image groups, each collaborative salient image group is The images in the image set are sequentially input into a pre-trained language model (such as the mPlug model) for model training. When the mPlug model receives the input image, it first extracts features from each image in the co-salient image group to obtain the image encoding features corresponding to each image. Then, in order to control the mPlug model to output the scene description the user wants, a suitable interactive text can be selected as the prompt input, and then feature extraction is performed on the interactive text to obtain the text encoding features.

[0084] Step S23: decoding the image coding features and text coding features corresponding to each image in the co-salient image group to obtain the scene structure knowledge description text corresponding to each image containing the relationship between different co-salient objects;

[0085] In this embodiment, after extracting features from each image and preset interactive text in the collaborative salient image group through the pre-trained language model, further, the image encoding features and text encoding features corresponding to each image in the collaborative salient image group are decoded to obtain the scene structure knowledge description text containing the relationship between different collaborative salient objects corresponding to each image in the collaborative salient image group. , the scene structure knowledge description text corresponding to all co-salient image groups can be expressed as .

[0086] Step S24: encoding each scene structure knowledge description text into text features respectively, and extracting keywords from the text features to obtain prompt keyword features corresponding to each scene structure knowledge description text;

[0087] In this embodiment, each scene structure knowledge description text Encode into text features, and extract keywords from each text feature to obtain the knowledge description text of each scene structure The corresponding prompt keyword features are used to describe the scene structure knowledge in sentences. Convert to prompt keyword features .

[0088] Step S25: mining the connections between different prompt keyword features to generate collaboratively significant text prompt words;

[0089] Next, explore the characteristics of different prompt keywords The connection between them generates a credible co-salient text prompt word .

[0090] Step S26: inputting the co-salient text prompt words into the segmentation model to convert the co-salient text prompt words into binary masks, thereby obtaining a co-salient detection model including historical co-salient detection results.

[0091] Furthermore, the salient text prompt words Input to the Segment Everything Model (SAM) to separate the co-salient textual cues Converted into a binary mask to obtain the historical co-significance detection result The co-saliency detection model is represents the co-saliency result of any image in a single co-salient image group. That is, the segmentation model (SAM) is used as the link between the co-salient textual cues. The bridge with the co-salient mask (i.e., binary mask) is to transform the co-salient textual prompt words Convert to a binary mask.

[0092] Specifically, step S22 extracts features from each image in the collaborative salient image group to obtain image coding features, and extracts features from the preset interactive text to obtain text coding features. Specifically, it may include: using an image feature extractor to extract features from each image in the collaborative salient image group to obtain image coding features, and converting the image coding features into vector representations to obtain image feature vectors; using a text feature extractor to extract features from the preset interactive text to obtain text coding features, and converting the text coding features into vector representations to obtain text feature vectors; wherein the image feature extractor and the text feature extractor are located in the pre-trained language model. In this embodiment, see Figure 3 As shown in Figure 1, the image feature extractor in the pre-trained language model (mPlug model) can be used to extract features from the images in the co-salient image group to obtain image coding features, and then the image coding features are converted into vector representations to obtain image feature vectors. ; Next, the text feature extractor in the pre-trained language model (such as the mPlug model) is used to extract features from the preset interactive text, such as what is Saliency in this image? (What is salient in this image?), to obtain text encoding features, and the text encoding features are converted into vector representations to obtain text feature vectors By converting the text encoding features into vectors, the interactive text can be converted from a computer-unprocessable form to a computer-processable form. Specifically, the process of compressing the interactive text into a feature vector can be expressed as:

[0093] ;

[0094] In the formula, text represents interactive text, It is a text feature extractor.

[0095] Specifically, in step S23, the image coding features and text coding features corresponding to each image in the collaboratively salient image group are decoded to obtain the scene structure knowledge description text corresponding to each image, which contains the relationship between different collaboratively salient objects. Specifically, it may include: combining the image feature vector and text feature vector corresponding to each image in the collaboratively salient image group to obtain a combined feature vector; inputting the combined feature vector into the first decoder located in the pre-trained language model to decode the combined feature vector to obtain the scene structure knowledge description text corresponding to each image, which contains the relationship between different collaboratively salient objects. In this embodiment, the image feature vectors corresponding to each image in the collaboratively salient image group are first decoded. and text feature vector The combined feature vector is then input into the decoder of the pre-trained language model to decode the combined feature vector and obtain the scene structure knowledge description text D corresponding to each image, which contains the relationship between different co-significant objects (i.e. Figure 3 The scene structure knowledge description text D can be specifically expressed as:

[0096] ;

[0097] In the formula, , I represents the input image, represents the image feature extractor, Represents a feature decoder.

[0098] Specifically, the process of acquiring scene structure knowledge description text of a single co-salient image group can be expressed as:

[0099] ;

[0100] In the formula, represents a large model for image understanding, represents the input image, and text represents interactive text. Represents the scene structure knowledge description text corresponding to the input image.

[0101] It should be pointed out that the scene structure knowledge description text not only covers the description of the salient target itself, but also includes the logical relationship between the salient target and the surrounding interference background. This relationship constitutes the scene structure knowledge.

[0102] Specifically, in step S24, each scene structure knowledge description text is encoded into a text feature, and keywords in the text features are extracted to obtain prompt keyword features corresponding to each scene structure knowledge description text. Specifically, it may include: using a text encoder to encode each scene structure knowledge description text to obtain text features, and linearly mapping the text features to obtain mapped features; using a multi-head self-attention mechanism to enhance the mapped features to obtain enhanced features; using a keyword extractor to extract keywords from the enhanced features to obtain multiple prompt keyword features corresponding to each scene structure knowledge description text. In this embodiment, a text encoder, such as a BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language representation model based on the Transformer architecture) text encoder, may be used to encode each scene structure knowledge description text respectively. Encode to describe the scene structure knowledge into text Convert to a computer-recognizable encoding form to obtain text features , text features Specifically, it can be expressed as:

[0103] ;

[0104] In the formula, A pre-trained BERT text encoder.

[0105] Next, the obtained text features Through linear mapping , , , and then use the Multi-Head Self-Attention mechanism to explore the connection between words in the sentence to strengthen the text features , the specific formula is:

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] In the formula, , , Three different linear layers representing queries, keys, and values, respectively. Represents multi-head self-attention.

[0111] Also, see Figure 4 As shown, Figure 4 The structure of multi-head self-attention is shown, including self-attention module, multi-layer perceptron (MLP), Add (feature fusion) layer and normalization layer, and the input is Q, K, V. In this embodiment, the text features can also be enhanced. To enhance, specifically, the text features can be enhanced Further through the fully connected layer, the specific formula is as follows:

[0112] ;

[0113] ;

[0114] In the formula, Represents enhanced text features After residual connection and layer normalization, the intermediate state features, Representation layer normalization, is the final enhanced feature after scene knowledge enhancement. represents a fully connected layer.

[0115] Furthermore, the enhanced features are input into the keyword extractor for keyword extraction to obtain the knowledge description text of each scene structure. Corresponding multiple prompt keyword features. This embodiment uses a multi-head self-attention mechanism to enhance the text features after linear mapping, and uses a keyword extractor to extract keywords from the enhanced features, which can make the obtained prompt keyword features more accurate and help improve the accuracy of real-time collaborative saliency detection.

[0116] Specifically, mining the connections between different prompt keyword features in step S25 to generate collaborative salient text prompt words may include: using a collaborative attention mechanism to mine the connections between different prompt keyword features to obtain collaborative text prompt features; decoding the collaborative text prompt features to obtain collaborative salient text prompt words. Figure 5 As shown in Figure 2, the co-attention mechanism can be used to connect different prompt keyword features to obtain the collaborative text prompt feature. , and then the collaborative text cue features are decoded through the BERT decoder to obtain the collaborative salient text cue words.

[0117] In a specific implementation, the collaborative attention mechanism is used to mine the connection between different prompt keyword features to obtain collaborative text prompt features, including: combining multiple prompt keyword features corresponding to each scene structure knowledge description text to obtain a keyword group; inputting the keyword group into different linear layers respectively, and multiplying the output results of different linear layers to obtain the attention affinity graph of the keyword group; extracting the first first number of values ​​in the attention affinity graph by row, and calculating the sum of the extracted first number of values ​​to obtain the first sum; calculating the ratio of the first sum to the first number to obtain the keyword weight, and multiplying the keyword weight by the prompt keyword features in the keyword group element by element to obtain multiple calculation results; extracting the first second number of values ​​in the multiple calculation results, and calculating the sum of the extracted second number of values ​​to obtain the second sum; calculating the ratio of the second sum to the second number to obtain the collaborative text prompt feature. In this embodiment, the multiple prompt keyword features corresponding to each scene structure knowledge description text can be combined to obtain a keyword group T, and then the keyword group T can be input into two different linear layers respectively, and the output results of the two linear layers can be multiplied to obtain the attention affinity graph of the keyword group T. The specific calculation formula is:

[0118] ;

[0119] In the formula, and Represents the linear mapping functions corresponding to two different linear layers, which are used to linearly map keyword features. represents the attention affinity graph, and k represents the total number of keyword features in the keyword group.

[0120] Next, see Figure 6 As shown, the first n maximum values ​​in the attention affinity graph are extracted row by row, and the sum of the first n maximum values ​​extracted is calculated to obtain the first sum, and the ratio of the first sum to n is calculated to obtain the keyword weight, that is, the maximum values ​​are summed and averaged to obtain the keyword weight. The specific calculation formula is:

[0121] ;

[0122] In the formula, It means taking the first n maximum values ​​in a vector of length j. It means that the mean of the vector with length 1 is output for the vector with length n. Indicates the keyword weight.

[0123] Furthermore, the keyword weight Multiply the prompt keyword features in the keyword group T element by element to obtain multiple calculation results, extract the first k maximum values ​​from the multiple calculation results, and calculate the sum of the extracted first k maximum values ​​to obtain the second sum value, and then calculate the ratio of the second sum value to k to obtain the collaborative text prompt feature , the specific calculation formula is:

[0124] ;

[0125] In the formula, Indicates a keyword group. represents element-wise multiplication, Represents the output collaborative text prompt feature.

[0126] It should be noted that, in the process of training the pre-trained language model and the segmentation model, it can also include: using real collaborative salient category text as a supervisory signal to supervise the learning process of the keyword extractor and the collaborative attention mechanism. In this embodiment, in order to ensure the accuracy of the extraction of collaborative salient text prompt words, in the process of training the pre-trained language model and the segmentation model, the real collaborative salient category text can be used as a supervisory signal to supervise the learning process of the keyword extractor and the collaborative attention mechanism.

[0127] In addition, in the process of training the pre-trained language model and the segmentation model, the following steps may be further included: obtaining co-salient text labels from the image dataset, and performing high-dimensional mapping on the co-salient text labels to obtain true co-prompt features; using the distance between the true co-prompt features and the co-text prompt features as the loss, and supervising the loss. Figure 7 As shown in the figure, during the model training process, the real collaborative prompt features obtained after the real collaborative salient text labels are compiled by BERT can be combined with the collaborative text prompt features Distance As the loss for supervision, the specific calculation formula is:

[0128] ;

[0129] In the formula, Representing predicted collaborative textual hint features , It represents the true collaborative prompt feature obtained after the true label is mapped by BERT, and k represents the dimension of the feature.

[0130] In addition, in order to facilitate subsequent calculations, the collaborative text prompt feature Converted into actual text, specifically, the collaborative text prompt features can be converted into actual text by BERT decoder Decoded into actual co-salient textual cues , the specific calculation formula is:

[0131] ;

[0132] In the formula, is the BERT decoder.

[0133] In this embodiment, the co-significant text prompt words are converted into binary masks to obtain a co-significant detection model containing historical co-significant detection results. Specifically, the co-significant text prompt words are feature encoded to obtain text feature encoding; feature encoded images in each co-significant image group to obtain image feature encoding; image feature encoding and corresponding text feature encoding are decoded to convert the co-significant text prompt words into binary masks to obtain historical co-significant detection results, so as to complete the training of the pre-trained language model and the segmentation model to obtain the co-significant detection model. Feature encoding is performed to obtain text feature encoding, and then feature encoding is performed on the images in each co-salient image group to obtain image feature encoding. The image feature encoding and the corresponding text feature encoding are decoded to convert the co-salient text prompt words into binary masks and obtain the corresponding historical co-salient detection results to complete the training of the entire co-salient detection model.

[0134] Specifically, feature encoding the co-salient text prompt words to obtain text feature encoding may include: inputting the co-salient text prompt words into a text prompt encoder to feature encode the co-salient text prompt words to obtain text feature encoding; correspondingly, feature encoding the images in each co-salient image group to obtain image feature encoding, including: inputting the images in each co-salient image group into an image encoder in sequence to feature encode each image in the co-salient image group to obtain image feature encoding; wherein the text prompt encoder and the image encoder are located in the segmentation model. In this embodiment, the co-salient text prompt words can be input into a text prompt encoder to feature encode the co-salient text prompt words to obtain text feature encoding; The images in each co-salient image group are input into the image encoder located in the segmentation model in turn, so as to perform feature encoding on each image in the co-salient image group and obtain image feature encoding.

[0135] Specifically, decoding the image feature code and the corresponding text feature code to convert the co-salient text prompt word into a binary mask to obtain the historical co-saliency detection result can include: respectively representing the image feature code and the corresponding text feature code by vectors to obtain a text prompt vector and an image code vector; combining the text prompt vector and the corresponding image code vector to obtain a combined vector; inputting the combined vector into a second decoder located in the segmentation model to decode the combined vector to obtain the historical co-saliency detection result including the co-saliency mask. In this embodiment, the text feature code and the image feature code are respectively compressed into a feature vector by the text prompt encoder and the image encoder, thereby converting the co-salient text prompt word from a computer-unprocessable form to a computer-processable form to obtain the corresponding text prompt vector. and the image encoding vector ; Next, the text prompt vector and the corresponding image encoding vector The combined vector is obtained by combining, and the combined vector is input into the decoder of the segmentation model to decode the combined vector to obtain a high-quality semantically guided collaborative saliency mask (i.e., a binary mask). The specific calculation formula is:

[0136] ;

[0137] In the formula, , ;in, represents the input image, represents the image encoder, Represents a text prompt encoder, is the feature decoder, and M is the mask result of a single image.

[0138] The mask results of all images in a single image group can be expressed as:

[0139] ;

[0140] In the formula, SAM represents the segmentation of all large models. represents the input image, where , represents the co-salient text prompt word, repeat the above model training process until all the co-salient image groups in step S21 have been subjected to co-saliency detection, and obtain all the co-saliency masks generated by the co-salient text prompt word through the SAM model, and obtain the final historical co-saliency detection result after post-processing .

[0141] Step S13: obtaining the current co-saliency detection result corresponding to the current image group output by the co-saliency detection model; wherein the current co-saliency detection result is a result obtained by extracting target text information related to the co-saliency between target images in the current image group based on a pre-trained language model, and guiding the segmentation model based on the target text information to segment the co-salient targets and backgrounds in the current image group. The current co-saliency detection result includes a mask result generated based on the target text information.

[0142] It should be pointed out that when the co-saliency detection model performs co-saliency detection on multiple target images in the current image group, it first extracts the text information related to the co-saliency between the target images in the current image group to obtain the target text information, and then guides the segmentation model based on the target text information to segment the co-salient targets and interference background in the current image group to obtain the current co-saliency detection result; wherein the current co-saliency detection result includes a mask result generated based on the target text information, that is, a set of co-saliency masks for each image in the current image group, and the co-saliency mask is a binary mask.

[0143] For details, see Figure 8 As shown, Figure 8The results of co-saliency detection on three groups of images using the co-saliency detection model proposed in this application are shown. From the co-saliency detection results, it can be seen that the first group is avocado, the second group is bird, and the third group is axe. It can be seen that this scheme can accurately extract the semantic information of the target in a complex and changing background, and obtain a more accurate mask result under the guidance of high-quality semantic information, thereby improving the accuracy of co-saliency detection.

[0144] It can be seen that the embodiment of the present application first obtains at least two target images to be detected to obtain a current image group; the current image group is input into the trained co-saliency detection model; the co-saliency detection model is a model obtained by training a pre-trained language model and a segmentation model based on an image data set containing co-saliency objects; the current co-saliency detection result corresponding to the current image group output by the co-saliency detection model is obtained; wherein the current co-saliency detection result is a result obtained by extracting target text information related to the co-saliency between target images in the current image group based on the pre-trained language model, and guiding the segmentation model based on the target text information to segment the co-saliency targets and background in the current image group, and the current co-saliency detection result includes a mask result generated based on the target text information. The present application pre-trains a pre-trained language model and a segmentation model based on an image data set containing co-salient objects to obtain a model for co-saliency detection. When co-saliency detection is required, the current image group to be detected is input into the model, thereby using the pre-trained language model to extract target text information related to the co-saliency between target images in the current image group, and guiding the segmentation model based on the extracted target text information to segment the co-salient targets and background in the current image group, thereby obtaining a co-saliency detection result including a mask result generated based on the target text information; the embodiment of the present application uses a pre-trained language model and a segmentation model to solve the problem of extracting co-saliency information from a cross-modal perspective, because the pre-trained language model can perform multi-modal interaction and establish cross-modal implicit semantic relationships. , and has the ability to understand image-text, so it can generate credible text scene descriptions for images, and the segmentation everything model has excellent instance segmentation effect. Therefore, through the combination of the pre-trained language model and the segmentation everything model, the difficult problem of image collaborative relationship extraction can be transformed into a simpler text collaborative relationship extraction problem, and high-quality saliency-related text information can improve the ability to distinguish between collaborative salient targets and interference backgrounds in collaborative saliency detection. In addition, using high-quality saliency-related text information to guide the segmentation everything model to segment collaborative salient targets can break through the limitation of semantically irrelevant binary masks as supervisory signals, which helps to alleviate the problem of limited representation learning caused by training scene limitations, thereby further improving the ability to recognize collaborative salient targets and interference backgrounds, and thus improving the accuracy of collaborative saliency detection.

[0145] Furthermore, the present application also discloses an electronic device. Fig. 9 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0146] Fig. 9 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the collaborative saliency detection method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0147] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0148] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0149] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and may be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs that can be used to complete the collaborative saliency detection method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0150] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned collaborative saliency detection method is implemented. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, and no further description will be given here.

[0151] Furthermore, an embodiment of the present application also discloses a computer program product, including a computer program / instruction, which implements the steps of the collaborative saliency detection method disclosed above when the computer program / instruction is executed by a processor.

[0152] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0153] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0154] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0155] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0156] The above is a detailed introduction to a collaborative saliency detection method, product, device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A collaborative saliency detection method, characterized in that: include: Acquire at least two target images to be detected to obtain a current image group; Inputting the current image group into a trained co-saliency detection model; the co-saliency detection model is a model obtained by training a pre-trained language model and a segmentation model based on an image dataset containing co-saliency objects; Obtain a current co-saliency detection result corresponding to the current image group output by the co-saliency detection model; wherein the current co-saliency detection result is a result obtained by extracting target text information related to the co-saliency between target images in the current image group based on the pre-trained language model, and guiding the segmentation model to segment the co-salient targets and backgrounds in the current image group based on the target text information, and the current co-saliency detection result includes a mask result generated based on the target text information.

2. The collaborative saliency detection method according to claim 1, characterized in that: Also includes: Acquire an image dataset containing co-salient objects, and group images in the image dataset according to different co-salient objects to obtain a plurality of co-salient image groups; Inputting the images in each of the collaborative salient image groups into a pre-trained language model in sequence to extract features from each of the images in the collaborative salient image groups to obtain image coding features, and extracting features from a preset interactive text to obtain text coding features; Decoding the image encoding features and the text encoding features corresponding to each image in the co-salient image group to obtain a scene structure knowledge description text corresponding to each image containing the relationship between different co-salient objects; Encoding each of the scene structure knowledge description texts into text features respectively, and extracting keywords from the text features to obtain prompt keyword features corresponding to each of the scene structure knowledge description texts; Mining the connections between the different prompt keyword features to generate collaboratively significant text prompt words; The co-salient text prompt words are input into a segmentation model to convert the co-salient text prompt words into binary masks, thereby obtaining the co-salient detection model including historical co-salient detection results.

3. The collaborative saliency detection method according to claim 2, characterized in that: The extracting features of each image in the collaborative salient image group to obtain image coding features, and extracting features of the preset interactive text to obtain text coding features, includes: Using an image feature extractor to extract features from each image in the collaborative salient image group to obtain image coding features, and converting the image coding features into vector representations to obtain image feature vectors; Using a text feature extractor to extract features from a preset interactive text to obtain text encoding features, and converting the text encoding features into vector representations to obtain text feature vectors; Wherein, the image feature extractor and the text feature extractor are located in the pre-trained language model.

4. The collaborative saliency detection method according to claim 3, characterized in that: The decoding of the image encoding features and the text encoding features corresponding to each image in the co-salient image group to obtain the scene structure knowledge description text corresponding to each image containing the relationship between different co-salient objects includes: Combining the image feature vector and the text feature vector corresponding to each image in the collaborative salient image group to obtain a combined feature vector; The combined feature vector is input into a first decoder located in the pre-trained language model to decode the combined feature vector to obtain a scene structure knowledge description text corresponding to each image containing the relationship between different co-saliency objects.

5. The collaborative saliency detection method according to claim 2, characterized in that: The encoding of each of the scene structure knowledge description texts into text features and extracting keywords from the text features to obtain prompt keyword features corresponding to each of the scene structure knowledge description texts includes: Using a text encoder to encode each of the scene structure knowledge description texts to obtain text features, and linearly mapping the text features to obtain mapped features; Using a multi-head self-attention mechanism to enhance the mapped features to obtain enhanced features; A keyword extractor is used to extract keywords from the enhanced features to obtain a plurality of prompt keyword features corresponding to each of the scene structure knowledge description texts.

6. The collaborative saliency detection method according to claim 5, characterized in that: The mining of the connection between the different prompt keyword features to generate collaboratively significant text prompt words includes: The connection between the different prompt keyword features is mined using the collaborative attention mechanism to obtain collaborative text prompt features; The collaborative text prompt feature is decoded to obtain a collaborative salient text prompt word.

7. The collaborative saliency detection method according to claim 6, characterized in that: The collaborative attention mechanism is used to mine the connections between the different prompt keyword features to obtain collaborative text prompt features, including: Combining multiple prompt keyword features corresponding to each of the scene structure knowledge description texts to obtain a keyword group; Inputting the keyword groups into different linear layers respectively, and multiplying the output results of different linear layers to obtain an attention affinity graph of the keyword group; Extracting a first number of values ​​in the attention affinity graph row by row, and calculating a sum of the first number of values ​​extracted to obtain a first sum; Calculating a ratio of the first sum value to the first number to obtain a keyword weight, and multiplying the keyword weight by the prompt keyword feature in the keyword group element by element to obtain a plurality of calculation results; Extracting first second number of values ​​from the plurality of calculation results, and calculating the sum of the extracted second number of values ​​to obtain a second sum; The ratio of the second sum value to the second quantity is calculated to obtain a collaborative text prompt feature.

8. The collaborative saliency detection method according to claim 6, characterized in that: In the process of training the pre-trained language model and the segmentation model, it also includes: The real co-salient category text is used as a supervision signal to supervise the learning process of the keyword extractor and the co-attention mechanism.

9. The collaborative saliency detection method according to claim 6, characterized in that: In the process of training the pre-trained language model and the segmentation model, it also includes: Acquire collaborative salient text labels from the image dataset, and perform high-dimensional mapping on the collaborative salient text labels to obtain true collaborative prompt features; The distance between the true collaborative prompt feature and the collaborative text prompt feature is used as a loss, and the loss is supervised.

10. The collaborative saliency detection method according to any one of claims 2 to 9, characterized in that: The converting the co-salient text prompt words into binary masks to obtain the co-salient detection model including historical co-salient detection results includes: Performing feature coding on the collaboratively significant text prompt words to obtain text feature coding; Performing feature coding on the images in each of the collaborative salient image groups to obtain image feature coding; The image feature code and the corresponding text feature code are decoded to convert the co-salient text prompt word into a binary mask to obtain a historical co-saliency detection result, so as to complete the training of the pre-trained language model and the segmentation model and obtain the co-saliency detection model.

11. The collaborative saliency detection method according to claim 10, characterized in that: The step of performing feature coding on the collaborative salient text prompt words to obtain text feature coding includes: Inputting the collaboratively significant text prompt words into a text prompt encoder to perform feature encoding on the collaboratively significant text prompt words to obtain text feature encoding; Accordingly, the performing feature coding on the images in each of the collaborative salient image groups to obtain image feature coding includes: Inputting the images in each of the co-salient image groups into an image encoder in sequence to perform feature encoding on each of the images in the co-salient image groups to obtain image feature encoding; Wherein, the text prompt encoder and the image encoder are located in the segmentation model.

12. The collaborative saliency detection method according to claim 11, characterized in that: The decoding of the image feature code and the corresponding text feature code to convert the co-salient text prompt word into a binary mask to obtain a historical co-salient detection result includes: Respectively performing vector representation on the image feature code and the corresponding text feature code to obtain a text prompt vector and an image code vector; Combining the text prompt vector and the corresponding image encoding vector to obtain a combined vector; The combined vector is input into a second decoder located in the segmentation model to decode the combined vector to obtain a historical co-saliency detection result including a co-saliency mask.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the collaborative saliency detection method according to any one of claims 1 to 12 is implemented.

14. An electronic device, characterized in that: The method comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the collaborative saliency detection method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the collaborative saliency detection method according to any one of claims 1 to 12 is implemented.

Images