A category weakly supervised RGB-D image salient object detection method
By combining category labels and pseudo-labels, and utilizing the training and update stages of the visual-language matching model and the image salient object detection model, the problem of insufficient accuracy in existing weakly supervised category methods is solved, and high-precision salient object detection in RGB-D images is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANHUI UNIV
- Filing Date
- 2022-04-27
- Publication Date
- 2026-05-08
AI Technical Summary
Existing categorically weakly supervised RGB-D image salient object detection methods rely on pre-training on large-scale classification datasets. The generated class activation maps are not accurate enough and cannot effectively focus on the entire object, resulting in insufficient model accuracy.
By combining category labels and pseudo-labels, and through the training and update phases, a visual-language matching model and an image salient object detection model are used to generate category similarity vectors and category-aware saliency maps, and update pseudo-labels to improve detection accuracy.
High-precision salient object detection in RGB-D images can be achieved without pixel-level ground truth labels, improving the model's detection accuracy and robustness.
Smart Images
Figure CN117036838B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and in particular to a weakly supervised method for detecting salient objects in RGB-D images. Background Technology
[0002] RGB-D image salient object detection aims to extract salient objects from RGB-D images by combining color and depth information. This extraction mimics the human visual attention mechanism, where only objects that attract human attention are considered salient. Traditional methods employ a fully supervised approach to train an RGB-D image salient object detection model, then input an RGB-D image to output a salient map. This method relies heavily on manually labeled data. To reduce reliance on manual annotation, methods utilizing weak supervision signals such as category information, bounding boxes, points, doodles, and counts have been proposed and have achieved some progress. Category information is readily available, and the recently proposed CapS dataset provides category information for RGB-D image salient object detection training datasets. Therefore, this invention focuses on using category information to achieve weak supervision in RGB-D image salient object detection. Existing weakly supervised methods, such as WSS (from the paper "Learning to Detect SalientObjects with Image-Level Supervision"), ASMO (from the paper "Weakly supervised salientobject detection using image labels"), MSW (from the paper "Multi-Source WeakSupervision for Saliency Detection"), and MFNet (from the paper "MFNet: Multi-Filter Directive Network for Weakly Supervised Salient Object Detection"), train classification models using ImageNet or COCO datasets. They then generate class activation maps from the features of the classification model, using these maps as pseudo-labels to supervise the training of the image salient object detection model. However, these methods require pre-training on large-scale classification datasets, and the generated class activation maps are not accurate enough, only highlighting class-related discriminative features rather than focusing on the entire object. Models trained using these as supervisory signals inevitably lack sufficient accuracy. Summary of the Invention
[0003] The technical problem to be solved by the present invention is to provide a salient object detection method for RGB-D images with weak category supervision, which uses category labels and pseudo-labels provided by unsupervised methods to achieve salient object detection in RGB-D images.
[0004] This invention discloses a category-weakly supervised RGB-D image salient object detection method, comprising a training phase and an update phase. In the training phase, pseudo-labels supervise the training of the image salient object detection model, outputting a saliency map. Simultaneously, category labels and pseudo-labels supervise the training of a visual-language matching model, outputting a category similarity vector and a category-aware saliency map. In the update phase, the category-aware saliency map and the pseudo-labels are weighted and summed to update the pseudo-labels. The update phase occurs during the training phase. In the testing phase, the category-weakly supervised RGB-D image salient object detection method tests any input RGB-D image using the trained image salient object detection model, outputting the final saliency map.
[0005] Furthermore, the aforementioned categorically weakly supervised RGB-D image salient object detection method is characterized in that, during the training phase, the input image x generates initial pseudo-labels pm using an unsupervised method, wherein the unsupervised method refers to the method described in the paper "An Innovative Salient Object Detection using Center-Dark Channel Prior"; the pseudo-labels pm supervise the image salient object detection model M. SOD The training outputs a saliency map m;
[0006] m = M SOD (x, pm)
[0007] Where function M SOD The first parameter of (·, ·) represents the image input of the model, the second parameter represents the supervision signal for model training, and the function returns the output of the model: the saliency map m;
[0008] Simultaneously, the visual-language matching model M is supervised by category label cls and pseudo-label pm. VL The training outputs a category similarity vector (csv) and a category-aware saliency map (cm).
[0009] [csv, cm] = M VL (x, t, cls, pm)
[0010] Where function M VLThe first parameter of (·, ·, ·, ·) represents the image input of the model, the second parameter represents the text input formed by all salient categories in the training set, the third parameter represents the model's supervision signal category label cls, and the fourth parameter represents the model's supervision signal pseudo-label pm. The function returns two outputs of the model: a category similarity vector csv and a category-aware saliency map cm. The second parameter remains constant throughout the training process.
[0011] Furthermore, the aforementioned category-weakly supervised RGB-D image salient object detection method is characterized in that, in the update stage, the category-aware salient map cm and the pseudo-label pm are weighted and summed based on the category similarity of the images under the mask, and the pseudo-label pm is updated using this result. The specific process is as follows:
[0012] First, the input image x is used to generate a category-aware saliency map mask image x under the mask of the category-aware saliency map cm and the pseudo-label pm. cm and pseudo-label mask image x pm ;
[0013] x j =Conv(x×Smooth(j))
[0014] Where j∈{cm,pm} represents a category-aware saliency map or pseudo-label, Smooth(·) refers to Gaussian smoothing operation, and Conv(·) refers to convolution operation;
[0015] Secondly, the category-aware saliency map mask image x cm and pseudo-label mask image x pm Input into the visual-language matching model M VL Perform the test and output the category-aware saliency map mask image x. cm and pseudo-label mask image x pm The corresponding category similarity vector csv j ;
[0016] [csv j cm j ] = M VL (x j ,t,-,-)
[0017] Where model M VL The third and fourth parameters are empty, indicating that this is a test and no supervision signal is needed. The second output cmj is ignored.
[0018] Then, based on the classification similarity indicated by the category label cls, a confidence score is generated between the category-aware saliency map cm and the pseudo-label pm. cm and Scorepm ;
[0019] [Score cm Score pm ] = softmax(csv cm [cls], csv pm [cls])
[0020] csv j [cls] refers to retrieving the category similarity vector CSV. j The value at the position of the category label cls, where softmax(·,·) represents the soffmax function;
[0021] Finally, using the confidence scores as weights, the category-aware saliency map cm and the pseudo-label pm are weighted and summed to form the updated pseudo-label pm.
[0022] pm←Score cm ×cm+Score pm ×pm
[0023] Furthermore, the categorical weakly supervised RGB-D image salient object detection method is characterized in that the update phase occurs during the training phase; after n training iterations, an update is performed once, balancing model performance and time cost, with n preferably being 3.
[0024] Furthermore, the aforementioned categorically weakly supervised RGB-D image salient object detection method is characterized in that the image salient object detection model M... SOD A U-Net structure is adopted for fusion of four-channel input of RGB-D images.
[0025] Furthermore, the aforementioned categorically weakly supervised RGB-D image salient object detection method is characterized in that the image salient object detection model M... SOD The loss function is defined as follows:
[0026]
[0027] Where loss ppa (·,·) refers to the pixel position-aware loss proposed in the paper F3Net: Fusion, Feedback and Focus for SalientObject Detection.
[0028] Furthermore, the aforementioned categorically weakly supervised RGB-D image salient object detection method is characterized in that the visual-language matching model M... VL By visual encoder ε vand language encoder ε t Composition; the visual encoder ε v and language encoder ε t The CLIP visual encoder and language encoder are employed; the CLIP visual encoder and language encoder are those proposed in the paper "Learning Transferable Visual Models From Natural Language Supervision"; the visual encoder ε v The input image x is encoded to form high-level features f4, which are then processed by global average pooling (GAP) to form global features. The two signals, after undergoing multi-head self-attention MHSA operation, form a high-level signal z and a global signal.
[0029] f4=ε v (x)
[0030]
[0031]
[0032] The language encoder ε t Encode all salient category texts in the training set to form category text features t;
[0033] t = ε t ([e1, e2, ..., e K ])
[0034] Among them, e k (k∈{1,…K}) represents the text representation of K salient categories;
[0035] The similarity between the category text feature t and the high-level signal z is calculated to form the category similarity matrix score;
[0036] score = Norm(z) · Norm(t) T
[0037] The category similarity matrix score and the high-level feature f4 are concatenated and then fed into the decoder to generate a category-aware saliency map cm; the decoder is a deconvolution operation that is concatenated from high level to low level.
[0038] cm=Decoder(Concatenation(f4,score))
[0039] Simultaneously, the categorical text features t are compared with the global signal. Perform similarity calculations to generate a category similarity vector (CVS).
[0040]
[0041] Furthermore, the aforementioned categorically weakly supervised RGB-D image salient object detection method is characterized in that the visual-language matching model M... VL The loss function is defined as follows:
[0042]
[0043] Where loss ppa (·,·) refers to the pixel location-aware loss proposed in the paper F3Net: Fusion, Feedback and Focus for SalientObject Detection. ce (·,·) represents the cross-entropy loss.
[0044] Compared with existing technologies, the beneficial effects of this invention are reflected in:
[0045] This invention provides a salient object detection method for RGB-D images with weak category supervision, which supervises the training of the model with category labels and pseudo-labels, without requiring pixel-level ground truth labels. Attached Figure Description
[0046] Figure 1 This is a schematic diagram of a category-weakly supervised RGB-D image salient object detection method according to the present invention;
[0047] Figure 2 This is a structural diagram of the visual-language matching model in a category-weakly supervised RGB-D image salient object detection method of the present invention;
[0048] The present invention will be further described below through specific embodiments and in conjunction with the accompanying drawings, but the embodiments of the present invention are not limited thereto. Detailed Implementation
[0049] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.
[0050] This embodiment presents a categorically weakly supervised RGB-D image salient object detection method, such as... Figure 1As shown, the method includes a training phase and an update phase. The training phase is supervised by pseudo-labels to train the image salient object detection model, outputting a saliency map. Simultaneously, the visual-language matching model is supervised by category labels and pseudo-labels, outputting a category similarity vector and a category-aware saliency map. The update phase uses a weighted sum of the category similarity of the images under the category-aware saliency map and the pseudo-label mask to update the pseudo-labels. The update phase occurs during the training phase. In the testing phase, the weakly supervised RGB-D image salient object detection method tests any input RGB-D image using the trained image salient object detection model, outputting the final saliency map.
[0051] Furthermore, the aforementioned categorically weakly supervised RGB-D image salient object detection method is characterized in that, during the training phase, the input image x generates initial pseudo-labels pm using an unsupervised method, wherein the unsupervised method refers to the method described in the paper "An Innovative Salient Object Detection using Center-Dark Channel Prior"; the pseudo-labels pm supervise the image salient object detection model M. SOD The training outputs a saliency map m;
[0052] m = M SOD (x, pm)
[0053] Where function M SOD The first parameter of (·, ·) represents the image input of the model, the second parameter represents the supervision signal for model training, and the function returns the output of the model: the saliency map m;
[0054] Simultaneously, the visual-language matching model M is supervised by category label cls and pseudo-label pm. VL The training outputs a category similarity vector (csv) and a category-aware saliency map (cm).
[0055] [csv, cm] = M VL (x, t, cls, pm)
[0056] Where function M VL The first parameter of (·, ·, ·, ·) represents the image input of the model, the second parameter represents the text input formed by all salient categories in the training set, the third parameter represents the model's supervision signal category label cls, and the fourth parameter represents the model's supervision signal pseudo-label pm. The function returns two outputs of the model: a category similarity vector csv and a category-aware saliency map cm. The second parameter remains constant throughout the training process.
[0057] Furthermore, the aforementioned category-weakly supervised RGB-D image salient object detection method is characterized in that, in the update stage, the category-aware salient map cm and the pseudo-label pm are weighted and summed based on the category similarity of the images under the mask, and the pseudo-label pm is updated using this result. The specific process is as follows:
[0058] First, the input image x is used to generate a category-aware saliency map mask image x under the mask of the category-aware saliency map cm and the pseudo-label pm. cm and pseudo-label mask image x pm ;
[0059] x j =Conv(x×Smooth(j))
[0060] Where j∈{cm,pm} represents a category-aware saliency map or pseudo-label, Smooth(·) refers to Gaussian smoothing operation, and Conv(·) refers to convolution operation;
[0061] Secondly, the category-aware saliency map mask image x cm and pseudo-label mask image x pm Input into the visual-language matching model M VL Perform the test and output the category-aware saliency map mask image x. cm and pseudo-label mask image x pm The corresponding category similarity vector csv j ;
[0062] [csv j cm j ] = M VL (x j ,t,-,-)
[0063] Where model M VL The third and fourth parameters are empty, indicating that this is a test and no supervisory signal is needed. The second output is cm. j Ignored;
[0064] Then, based on the classification similarity indicated by the category label cls, a confidence score is generated between the category-aware saliency map cm and the pseudo-label pm. cm and Score pm ;
[0065] [Score cm Score pm ] = softmax(csv cm [cls], csv pm [cls])
[0066] csvj [cls] refers to retrieving the category similarity vector CSV. j The value at the position of the category label cls, where softmax(·,·) represents the soffmax function;
[0067] Finally, using the confidence scores as weights, the category-aware saliency map cm and the pseudo-label pm are weighted and summed to form the updated pseudo-label pm.
[0068] pm←Score cm ×cm+Score pm ×pm
[0069] Furthermore, the categorical weakly supervised RGB-D image salient object detection method is characterized in that the update phase occurs during the training phase; after n training iterations, an update is performed once, balancing model performance and time cost, with n preferably being 3.
[0070] Furthermore, the aforementioned categorically weakly supervised RGB-D image salient object detection method is characterized in that the image salient object detection model M... SOD A U-Net structure is adopted for fusion of four-channel input of RGB-D images.
[0071] Furthermore, the aforementioned categorically weakly supervised RGB-D image salient object detection method is characterized in that the image salient object detection model M... SOD The loss function is defined as follows:
[0072]
[0073] Where loss ppa (·,·) refers to the pixel position-aware loss proposed in the paper F3Net: Fusion, Feedback and Focus for SalientObject Detection.
[0074] Furthermore, the aforementioned categorically weakly supervised RGB-D image salient object detection method is characterized in that the visual-language matching model M... VL ,like Figure 2 As shown, by the visual encoder ε v and language encoder ε t Composition; the visual encoder ε v and language encoder ε tThe visual encoder and language encoder of CLIP are adopted; the visual encoder and language encoder of CLIP are the visual encoder and language encoder proposed in the paper Learning Transferable Visual Models From Natural Language Supervision; the visual encoder ε v The input image x is encoded to form high-level features f4, which are then processed by global average pooling (GAP) to form global features. The two signals, after undergoing multi-head self-attention MHSA operation, form a high-level signal z and a global signal.
[0075] f4=ε v (x)
[0076]
[0077]
[0078] The language encoder ε t Encode all salient category texts in the training set to form category text features t;
[0079] t = ε t ([e1, e2, ..., e K ])
[0080] Among them, e k (k∈{1,…K}) represents the text representation of K salient categories;
[0081] The similarity between the category text feature t and the high-level signal z is calculated to form the category similarity matrix score;
[0082] score = Norm(z) · Norm(t) T
[0083] The category similarity matrix score and the high-level feature f4 are concatenated and then fed into the decoder to generate a category-aware saliency map cm; the decoder is a deconvolution operation that is concatenated from high level to low level.
[0084] cm=Decoder(Concatenation(f4,score))
[0085] Simultaneously, the categorical text features t are compared with the global signal. Perform similarity calculations to generate a category similarity vector (CVS).
[0086]
[0087] Furthermore, the aforementioned categorically weakly supervised RGB-D image salient object detection method is characterized in that the visual-language matching model M... VL The loss function is defined as follows:
[0088]
[0089] Where loss ppa (·,·) refers to the pixel position-aware loss proposed in the paper F3Net: Fusion, Feedback and Focus for SalientObject Detection, and lossce(·,·) is the cross-entropy loss.
[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A categorically weakly supervised method for salient object detection in RGB-D images, characterized in that, Includes the training phase and the update phase; The training phase is performed by training a pseudo-label-supervised image salient object detection model to output a salient map; During the training phase, the input image x is processed using a conventional image processing method based on the center-dark channel prior to generate initial pseudo-labels. ; Simultaneously, the visual-language matching model is trained under the supervision of category labels and pseudo-labels to output category similarity vectors and category-aware saliency maps; In the update phase, the input image is first divided into two sets of mask images under the category-aware saliency map mask and the pseudo-label mask, respectively. Then, the two sets of mask images are input into the visual-language matching model to obtain the corresponding category similarity vector. The confidence scores of the category-aware saliency map and the pseudo-label are obtained according to the category label. Finally, the category-aware saliency map and the pseudo-label are weighted and summed to update the pseudo-label. In the process of obtaining confidence scores, based on the category label, the corresponding positions in the category similarity vectors are indexed from the two sets of category similarity vectors respectively, the values at those positions are read and extracted, the two sets of values are normalized, and the two sets of probability values are finally output as the confidence scores of the category-aware saliency map and the pseudo-label confidence scores respectively. The update phase occurs during the training phase; During the testing phase, the trained image salient object detection model is used to test any input RGB-D image and output the final salient map.
2. The method for salient object detection in RGB-D images with weak category supervision according to claim 1, characterized in that, During the training phase, pseudo-labels Supervise the image salient object detection model Training, outputting saliency maps ; ； Where the function The first parameter represents the image input to the model, the second parameter represents the supervision signal for model training, and the function returns the model's output: saliency map. ; Simultaneously, by category labels Pseudo-labels Supervised vision-language matching model The training outputs a class similarity vector. and category-aware saliency maps ; ； Where the function The first parameter represents the image input to the model, the second parameter represents the text input formed by all salient categories in the training set, and the third parameter represents the class labels of the supervision signal for the model. The fourth parameter represents the pseudo-label of the model's supervision signal. The function returns two outputs of the model: a class similarity vector. and category-aware saliency maps The second parameter remains constant throughout the entire training process.
3. The method for salient object detection in RGB-D images with weak category supervision according to claim 1, characterized in that, In the update phase, category-aware saliency maps are utilized. Mask and pseudo-tag The two sets of category similarity vectors of the image under the mask are weighted and summed to form a category-aware saliency map. and pseudo-tags Update the pseudo-tags based on the results. The specific process is as follows: First, input an image. In category-aware saliency maps Masks and pseudo-tags Under the mask, category-aware saliency map mask images are formed respectively. and pseudo-label mask image ; ； in This represents a category-aware saliency map or pseudo-label. This refers to Gaussian smoothing. This refers to the convolution operation; Secondly, the category-aware saliency map mask image and pseudo-label mask image Feed into the vision-language matching model Perform the test and output the category-aware saliency map mask image. and pseudo-label mask image Corresponding category similarity vector ; ； The model The third and fourth parameters are empty, indicating that this is a test and no supervisory signal is needed. The second output... Ignored; Then, based on category labels The classification similarity of the indicators is used to form a category-aware saliency map. With pseudo-tags confidence score and ; ； in This refers to retrieving the category similarity vector. Category Labels The value at the location, This represents the softmax function; Finally, using the confidence scores as weights, the category-aware saliency maps are summed using weighted averages. and pseudo-tags This creates newer pseudo-tags. ,have: 。 4. The method for salient object detection in RGB-D images with weak category supervision according to claim 1, characterized in that, The update phase occurs during the training phase; after experiencing After one training iteration, perform one update to weigh model performance against time cost. Take 3.
5. The method for salient object detection in RGB-D images with weak category supervision according to claim 2, characterized in that, The image salient object detection model A U-Net structure is adopted for fusion of four-channel input of RGB-D images.
6. The method for salient object detection in RGB-D images with weak category supervision according to claim 2, characterized in that, The image salient object detection model The loss function is defined as follows: ； in This represents the pixel location-aware loss function, used to calculate the pixel location matching loss between the saliency map and the pseudo-label.
7. The method for salient object detection in RGB-D images with weak category supervision according to claim 2, characterized in that, The visual-language matching model By visual encoder and language encoder Composition; the visual encoder and language encoder The visual encoder and language encoder employ CLIP; the visual encoder Encoding input image Forming high-level characteristics Global features are formed after global average pooling (GAP) operation. The two signals, after undergoing multi-head self-attention MHSA operation, form a high-level signal. and global signals ; ； ； ； The language encoder Encode all salient category texts in the training set to form category text features. ; ； in, express Text representation of 1 significant category; Category text features signals from higher levels Perform similarity calculations to form a category similarity matrix. ; ； The category similarity matrix With the aforementioned high-level features After the concatenation operation, the data is fed into the decoder to generate a category-aware saliency map. The decoder is a deconvolution operation that cascades from high to low layers. ； At the same time, the category text features With global signals Perform similarity calculations to generate category similarity vectors. ; 。 8. The method for salient object detection in RGB-D images with weak category supervision according to claim 2, characterized in that, The visual-language matching model The loss function is defined as follows: ； in This represents the pixel location-aware loss function, used to calculate the pixel location matching loss between the saliency map and the pseudo-label. It is cross-entropy loss.