A method and system for image saliency detection based on multi-source weak supervision

Abstract

The present invention provides a method and system for image saliency detection based on multi-source weak supervision, relating to the field of image processing technology. The method comprises: annotating RGB images based on text labels and point labels to generate pseudo labels; utilizing different encoders in a constructed salient target detection model to extract features from the RGB image and depth image, respectively; fusing the extracted features hierarchically to obtain multiple hierarchical image features; generating edge features based on the multiple hierarchical image features; and decoding the multiple hierarchical image features in combination with the edge features to obtain a salient target image. By combining the advantages of multiple weak labels and giving full play to the complementary role of depth information to RGB information, the salient targets detected by the salient target detection model are made to highlight the filamentous edge structure.

Description

Technical Field

[0001] The present invention mainly relates to the field of image processing technology, and in particular to an image saliency detection method and system based on multi-source weak supervision. Background Art

[0002] Salient object detection (SOD) aims to locate and segment the objects that most attract human attention in a scene. Existing SOD techniques, which rely on appearance information and leverage multimodal information to highlight objects, have made significant progress. Image SOD with depth information (RGB-D SOD) has become a mainstream technique. With the development of deep learning, fully supervised RGB-D SOD can achieve high-precision object detection, but it relies heavily on pixel-level annotations, which are cumbersome and expensive to obtain. Consequently, SOD methods using weakly supervised training have emerged.

[0003] In current mainstream weakly supervised RGB-D SOD research, using single-source weak annotations as supervision leads to problems such as weak pixel richness in training samples, poor structural integrity of salient objects, and weak prominence of salient targets. Therefore, it is clear that a single weak annotation is difficult to complete the SOD task. Using multiple weak annotations as supervision also leads to the problem of conflicting annotation characteristics, which ultimately leads to the degradation of the advantages of each annotation and the lack of salient prominence in the output results, both in terms of detail and overall performance. Since the input information of the RGB-D SOD task is RGB images and depth images, most weakly supervised RGB-D SOD tasks simply process both of them in the same backbone network. This inevitably leads to the final result treating salient targets and the surrounding objects as salient targets. Therefore, there is an urgent need to address the problems of poor structural integrity and weak prominence of salient targets caused by weak labels. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an image saliency detection method and system based on multi-source weak supervision in response to the shortcomings of the existing technology.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] A method for image saliency detection based on multi-source weak supervision includes the following steps:

[0007] Import RGB images, depth images, text labels, and point labels;

[0008] Matching the RGB image with the text label to obtain an initial pseudo label, and optimizing the initial pseudo label based on the point label to obtain a pseudo label;

[0009] Constructing a salient object detection model, the salient object detection model including a dual branch of a CNN encoder and a Swin-Transformer V2 encoder, and a hybrid attention module, an edge constraint module, and a progressive decoder sequentially connected to the dual branches;

[0010] Performing depth feature extraction on the depth image through the CNN encoder to obtain a depth feature set, performing channel feature extraction on the RGB image through the Swin-Transformer V2 encoder to obtain a multi-scale feature set, fusing the depth feature set and the multi-scale feature set through the hybrid attention module to obtain a plurality of different hierarchical image features, extracting edge features from the plurality of hierarchical image features through the edge constraint module, and decoding the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a salient target image;

[0011] Performing edge extraction on the salient target image using the edge constraint module to obtain edge supervision features, performing loss calculation on the edge features, the edge supervision features, the salient target image, and the pseudo-label to obtain a total loss, and optimizing the salient target detection model using the total loss to obtain an optimized salient target detection model;

[0012] The optimized salient target detection model is used to detect the RGB image to be detected and the depth image to be detected to obtain a target salient target image.

[0013] Another technical solution of the present invention to solve the above technical problems is as follows:

[0014] An image saliency detection system based on multi-source weak supervision, comprising:

[0015] Import unit, used to import RGB images, depth images, text labels and point labels;

[0016] a labeling unit, configured to match the RGB image with the text label to obtain an initial pseudo label, and optimize the initial pseudo label based on the point label to obtain a pseudo label;

[0017] A construction unit for constructing a salient object detection model, wherein the salient object detection model includes a dual branch of a CNN encoder and a Swin-Transformer V2 encoder, and a hybrid attention module, an edge constraint module, and a progressive decoder sequentially connected to the dual branches;

[0018] A prediction unit is configured to perform depth feature extraction on the depth image through the CNN encoder to obtain a depth feature set, perform channel feature extraction on the RGB image through the Swin-Transformer V2 encoder to obtain a multi-scale feature set, fuse the depth feature set and the multi-scale feature set through the hybrid attention module to obtain a plurality of different hierarchical image features, extract edge features from the plurality of hierarchical image features through the edge constraint module, and decode the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a salient target image;

[0019] an optimization unit, configured to perform edge extraction on the salient target image using the edge constraint module to obtain edge supervision features, perform loss calculation on the edge features, the edge supervision features, the salient target image, and the pseudo-label to obtain a total loss, and optimize the salient target detection model using the total loss to obtain an optimized salient target detection model;

[0020] The application unit is configured to detect the RGB image to be detected and the depth image to be detected by using the optimized salient target detection model to obtain a target salient target image.

[0021] The beneficial effects of the present invention are: it utilizes the multi-source combination of text information and image information to generate pseudo labels, making up for the inherent defects of weak labels, so that the model can learn the structural information of significant objects from annotations. By using two different backbone networks, the CNN encoder and the Swin-Transformer V2 encoder, different types of images are individually extracted for features, thereby avoiding the feature information containing too much interference noise, and the required features can be extracted in a targeted manner. The edge features of the image are extracted by the edge constraint module, and the significant features can be identified more clearly. The features of each level are decoded by a progressive decoder to identify all the features that show significance from the features of each level, so as to work together to enhance the feature decoding performance of the model. A variety of loss functions are mixed to guide the training of the model, so that the model can detect more accurate results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of an image saliency detection method based on multi-source weak supervision provided by an embodiment of the present invention;

[0023] Figure 2 The RGB image provided by the embodiment of the present invention;

[0024] Figure 3 A depth image provided by an embodiment of the present invention;

[0025] Figure 4A salient target image provided by an embodiment of the present invention;

[0026] Figure 5 A structural diagram of a salient object detection model provided by an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of a pseudo-label generation process provided by an embodiment of the present invention;

[0028] Figure 7 A structural diagram of a CNN encoder provided in an embodiment of the present invention;

[0029] Figure 8 A structural diagram of the Swin-Transformer V2 encoder provided in an embodiment of the present invention;

[0030] Figure 9 A structural diagram of a hybrid attention module provided in an embodiment of the present invention;

[0031] Figure 10 A structural diagram of an edge constraint module provided in an embodiment of the present invention;

[0032] Figure 11 A structural diagram of a progressive decoder provided by an embodiment of the present invention;

[0033] Figure 12 A unit block diagram of an image saliency detection system based on multi-source weak supervision provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0035] Salient object detection models typically require a large number of pixel-level labeled samples for training. However, due to the high cost of strong supervision sources, saliency detection models based on weak supervision sources (such as classification labels and summary labels) have become a research hotspot. By integrating information from multiple weakly supervised sources, the reliance on high-quality labeled data is reduced, thereby lowering the labeling cost and improving the model's generalization ability. Multi-source weakly supervised learning uses partial, imprecise, or otherwise inaccurate information for training. This information is easier to obtain than manually labeled data, thus reducing the labeling cost and finding applications in scenarios where labeled data is difficult to obtain or expensive to label.

[0036] In the weakly supervised RGB-D SOD task, using single-source weak annotations as supervision presents several serious problems, namely, the weak richness of pixel training samples, poor structural integrity of salient objects, and weak prominence of salient targets. Due to the weak richness of pixel training samples, single-source weak annotations cannot provide enough important pixel samples for robust saliency perception learning. The poor structural integrity of salient objects prevents the RGB-D SOD model from directly learning the structural information of salient objects from the annotations. The weak prominence of salient targets may lead to the highlighting of background objects, which goes against the original intention of the SOD task. Using multi-source annotations as supervision leads to the problem of conflicting annotation characteristics, which ultimately leads to the degradation of the advantages of each annotation, and the model output lacks outstanding performance in both details and overall performance. Since the input information of the RGB-D SOD task is RGB images and depth images, which are essentially different, most current weakly supervised RGB-D SOD tasks simply put the RGB images and depth images into the same backbone, which will inevitably lead to the extraction of a lot of noise information, making it difficult to fully express the features of the depth image, and even weakening the detailed information of the RGB image features. As a result, the final result regards the salient target and the objects around the salient target as salient targets, which also goes against the original intention of the SOD task.

[0037] like Figure 1-Figure 5 As shown, an embodiment of the present invention provides an image saliency detection method based on multi-source weak supervision, comprising the following steps:

[0038] Import RGB images, depth images, text labels, and point labels;

[0039] Matching the RGB image with the text label to obtain an initial pseudo label, and optimizing the initial pseudo label based on the point label to obtain a pseudo label;

[0040] Constructing a salient object detection model, the salient object detection model including a dual branch of a CNN encoder and a Swin-Transformer V2 encoder, and a hybrid attention module, an edge constraint module, and a progressive decoder sequentially connected to the dual branches;

[0041] Performing depth feature extraction on the depth image through the CNN encoder to obtain a depth feature set, performing channel feature extraction on the RGB image through the Swin-Transformer V2 encoder to obtain a multi-scale feature set, fusing the depth feature set and the multi-scale feature set through the hybrid attention module to obtain a plurality of different hierarchical image features, extracting edge features from the plurality of hierarchical image features through the edge constraint module, and decoding the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a salient target image;

[0042] Performing edge extraction on the salient target image using the edge constraint module to obtain edge supervision features, performing loss calculation on the edge features, the edge supervision features, the salient target image, and the pseudo-label to obtain a total loss, and optimizing the salient target detection model using the total loss to obtain an optimized salient target detection model;

[0043] The optimized salient target detection model is used to detect the RGB image to be detected and the depth image to be detected to obtain a target salient target image.

[0044] It should be understood that RGB-D images include RGB images and depth images. RGB images are three-channel color images, while depth images contain image channels that contain information related to the surface distance of objects in the viewpoint scene. The channels themselves are similar to grayscale images, and each pixel value is the actual distance to the object as measured by the sensor.

[0045] In an embodiment of the present invention, in order to address the problems of poor structural integrity and weak prominence of salient targets caused by weak labels in weak supervision, the present invention combines text labels and point labels to complete the construction of weakly supervised pseudo labels, and uses an asymmetric dual-branch backbone to extract features according to the different characteristics of the input images. In addition, low-level features are combined with high-level features to generate edge features. At the same time, the edges of the salient target images predicted by the model are used as a guide, and the model is optimized using a loss function to improve the quality of the salient target map and edge map output by the model, thereby improving the model detection accuracy.

[0046] Preferably, if Figure 6 As shown, the matching of the RGB image and the text label to obtain an initial pseudo label includes:

[0047] Perform feature extraction on the RGB image and the text label using a CLIP model to obtain encoded attention weights, image features, and text features;

[0048] The image features and the text features are mapped using the Grad-CAM algorithm to obtain a category activation map, including: calculating the probability distribution of the image features and the text features using a softmax function to obtain a foreground class score and a background class score, and calculating the foreground class score and the background class score using a weight calculation expression to obtain multiple pixel point category weights, wherein the weight calculation expression is:

[0049]

[0050] in, is the category weight of the foreground class c in the kth channel in the last feature layer A of the CLIP model, Z is the total pixel value, and y c is the foreground pixel score (i.e., the score corresponding to the pixel point of foreground class c in the forward propagation of the CLIP model), is the pixel coordinate (i, j) on the kth channel in the last feature layer A of the CLIP model, s c is the foreground score, y c′ is the background pixel score (i.e., the score corresponding to the background pixel in the forward propagation of the CLIP model), s c′ is the background score, c is the foreground score, and c′ is the total score (including foreground and background scores. Since c′≠c, c′ is the background score).

[0051] is the gradient information of the foreground class back-propagated on the last feature layer A of the CLIP model, The gradient information of the background class backpropagated on the last feature layer A of the CLIP model;

[0052] Perform weighted summation on the category weights of the plurality of pixel points to obtain an initial category activation map, perform mask extraction on the initial category activation map to obtain a category mask, calculate the encoding attention weight, the initial category activation map, and the category mask using a weight fusion expression to obtain a category activation map, and the weight fusion expression is:

[0053]

[0054] Among them, M t is the class activation map, B c is the category mask, W at To encode the attention weight, M c is the initial category activation map, vec(·) is the matrix vectorization, and ⊙ is the element-by-element multiplication;

[0055] The pixels of the class activation map are filtered according to the set filtering parameters to obtain initial pseudo labels.

[0056] Specifically, the expression of the softmax function is:

[0057]

[0058] Among them, s c is the foreground score, exp(·) is the exponential function, y c is the foreground pixel fraction, y c′ is the total category pixel score, c′ is the total category, c′={1,...,c,...,N}, N is the number of categories.

[0059] The weighted sum of the multiple pixel point category weights is performed using a weighted expression, and the weighted expression is:

[0060]

[0061] Among them, ReLU(·) is the activation process.

[0062] The processing process of the weight fusion expression is as follows: the initial category activation map M of the foreground target class c Thresholding is performed, and the initial category activation map M c Find the connected areas and get the minimum rectangular bounding box covering these connected areas to get the category mask B c , and then compared with the encoded attention weight W output by the multi-head attention layer of the image encoder in the CLIP model at Combined, encoding attention weight W at For pixel-by-pixel activation of the initial category map M c Optimize.

[0063] Since the category activation map is essentially the confidence of each pixel in the image about the category, a hyperparameter λ (i.e., filtering parameter) is set to ignore pixels with too low confidence and obtain the initial pseudo label, which is expressed as:

[0064]

[0065] Among them, U is the initial pseudo label, (i, j) is the category activation map M t The coordinates of the pixel in the .

[0066] It should be understood that the CLIP model (or CLIP model) learns the matching relationship between images and text by comparing them, placing relevant text features and image features adjacent in space and irrelevant ones further apart. This captures the deep semantic connection between images and text, enabling cross-modal understanding. In this invention, the CLIP model is a pre-trained large network model.

[0067] The Grad-CAM algorithm is used to perform gradient-weighted class activation mapping. The importance of feature maps is determined by calculating the gradient of the feature map with respect to the CLIP model output, that is, class activation mapping is performed on the feature map. The specific steps include: obtaining the category probability of the model output from the forward propagation of the CLIP model (that is, the CLIP model performs the matching processing process); calculating the gradient of the output category probability with respect to the feature map based on the target category (that is, the gradient of back propagation); and weighted superposition of each gradient with the corresponding feature map to obtain a Grad-CAM image (that is, a category activation map).

[0068] In an embodiment of the present invention, text labels and RGB images are fed into a CLIP model to extract text and image features. The Grad-CAM technique is used to combine the two to output a class activation map. The gradient information is backpropagated back to the CLIP model. The attention weight map output by the multi-head attention mechanism in the image encoder of the CLIP model is combined with the class mask extracted from the class activation map to generate a class-aware attention weight map. This weight map is used to refine the class activation map into an initial pseudo-label pixel by pixel.

[0069] Preferably, if Figure 6 As shown, the point labels include foreground target point images and background target point images;

[0070] The optimizing the initial pseudo-label based on the point label to obtain the pseudo-label includes:

[0071] Performing feature extraction on the RGB image using a Sobel algorithm to obtain a rough edge image, and performing pixel filling processing on the rough edge image according to the foreground target point image using a FloodFill algorithm to obtain a foreground target activation image;

[0072] The foreground target activation image and the initial pseudo-label are added to obtain a background target image, the background target image is pixel-filled according to the background target point image using the FloodFill algorithm to obtain a background target activation image, and the background target activation image is refined using the conditional random field algorithm to obtain a pseudo-label.

[0073] Specifically, the process of obtaining the foreground target activation image is expressed as:

[0074] P f =F(S,E),

[0075] Among them, P f is the foreground target activation image, S is the foreground target point image, E is the rough edge image, and F(·) is the Flood Fill algorithm.

[0076] The process of obtaining pseudo labels based on the foreground target activation image is expressed as:

[0077] P = CRF(F(SS b ,P f +U)),

[0078] Among them, P is the pseudo label, S b is the background target point image, U is the initial pseudo label, CRF(·) is the conditional random field algorithm, F(·) is the Flood Fill algorithm, SS b Indicates the subtraction of the foreground target point image and the background target point image.

[0079] It should be understood that the Flood Fill algorithm is a basic algorithm for image processing, used to fill connected areas. Usually starting from a set pixel point, the algorithm fills the pixels adjacent to the set pixel point until a boundary or other specified conditions are encountered. Using the Flood Fill algorithm to expand the entire image, a complete salient target map is obtained, while also suppressing non-salient targets to obtain refined pseudo-labels. The Conditional Random Field (CRF) algorithm is used to refine the segmentation results of the pseudo-labels.

[0080] In an embodiment of the present invention, an image is annotated with point labels, and a Flood Fill algorithm is used to activate the point-annotated area. Then, the foreground activation area is used to supplement the insufficient target in the initial pseudo-label, and the background activation area is used to suppress the excess noise in the initial pseudo-label, thereby obtaining a pseudo-label close to the true value image to guide the training of the salient target detection model.

[0081] High-quality pseudo labels generated from text labels and point labels are used as supervisory signals to guide the training of the salient object detection model, thereby enhancing the model's ability to depict the filamentous structure of salient objects and achieving accurate detection of salient objects in RGB-D images (RGB images and depth images).

[0082] Preferably, if Figure 7 As shown, the depth feature extraction is performed on the depth image by the CNN encoder to obtain a depth feature set, including:

[0083] The CNN encoder includes a point-by-point convolution layer, a standard normalization layer, an activation function layer, a first common convolution block and a plurality of depth-separable convolution blocks connected in series, wherein the first common convolution block is connected to the first depth-separable convolution block, the remaining depth-separable convolution blocks are connected in sequence, and the last depth-separable convolution block is connected to the point-by-point convolution layer, the standard normalization layer and the activation function layer in sequence;

[0084] Performing a convolution operation on the depth image through the first common convolution block to obtain common convolution features;

[0085] The first depth-wise separable convolution block performs a convolution operation on the common convolution feature to obtain a first-level depth feature, and the second depth-wise separable convolution block performs a convolution operation on the first depth feature to obtain a second-level depth feature, and so on, until the last depth-wise separable convolution block performs a convolution operation on the output of the previous depth-wise separable convolution block to obtain a new-level depth feature;

[0086] Performing a point-by-point convolution operation on the new-level depth feature through the point-by-point convolution layer, normalizing the new-level depth feature after the point-by-point convolution through the standard normalization layer, and activating the normalized new-level depth feature through the activation function layer to obtain the optimal level depth feature;

[0087] The output of each depth-separable convolution block and the optimal level depth feature constitute a depth feature set (ie, the depth feature set is composed of multiple levels of depth features).

[0088] Specifically, in the process of extracting depth information features, the CNN-base encoder (i.e., CNN encoder) first uses a backbone consisting of a 1×1 convolutional layer (denoted as Conv 1) and two 3×3 convolutional layers (denoted as Conv 3) to extract depth features for the input depth image D. These layers (1×1 convolutional layer and two 3×3 convolutional layers) are equipped with standard normalization (BN) and ReLU activation functions. After processing, ordinary convolution features are obtained. Expressed as:

[0089]

[0090] The learned deep features are then fed into three stages (i.e., depthwise separable convolution blocks) in sequence. For each stage, the first depthwise separable convolution block performs a 3×3 depthwise (DW) convolution and a 1×1 pointwise (PW) convolution along with BN and ReLU activation functions on the normal convolution features. After processing, the first-level deep features are obtained The second depth-wise separable convolutional block performs a deep convolution on the first-level deep features. After processing, the second-level deep features are obtained The third depth-wise separable convolutional block performs a deep convolution on the second-level deep features. After processing, the third-level deep features are obtained (i.e., new level depth features), expressed as:

[0091]

[0092] The third level deep features are processed by a layer of 1×1 point (PW) convolution and BN and ReLU. Perform feature extraction to obtain the fourth-level deep features (i.e., optimal level depth feature), expressed as:

[0093]

[0094] Among them, ReLU is the activation function, BN(·) is the standard normalization process, Conv2d i (·) is a 2D image convolution operation, DWConv3 is a 3×3 depth convolution operation, and PWConv1(·) is a 1×1 point-by-point convolution operation.

[0095] The first-level deep features extracted by outputting multiple depth-separable convolutional blocks Second level deep features The third level deep features And the fourth-level deep features extracted after point-by-point convolution operation Composition of deep feature set F d for

[0096] Specifically, the CNN encoder includes a first point-by-point convolution layer, a standard normalization layer, an activation function layer, a first ordinary convolution block and three depth-separable convolution blocks; the first ordinary convolution block includes a first 1×1 ordinary convolution layer, a first standard normalization layer and a first activation function layer, a first 3×3 ordinary convolution layer, a second standard normalization layer and a second activation function layer, a second 3×3 ordinary convolution layer, a third standard normalization layer and a third activation function layer connected in sequence.

[0097] The step of performing a convolution operation on the depth image by using the first common convolution block to obtain common convolution features includes:

[0098] Performing a normal convolution operation on the depth image through the first 1×1 normal convolution layer to obtain a first normal convolution feature, normalizing the first normal convolution feature through the first standard normalization layer to obtain a first normalized feature, and activating the first normalized feature through the first activation function layer to obtain a first activated feature;

[0099] Performing a normal convolution operation on the first activation feature through the first 3×3 normal convolution layer to obtain a second normal convolution feature, normalizing the second normal convolution feature through the second standard normalization layer to obtain a second normalized feature, and activating the second normalized feature through the second activation function layer to obtain a second activation feature;

[0100] Performing an ordinary convolution operation on the second activation feature through the second 3×3 ordinary convolution layer to obtain a third ordinary convolution feature, normalizing the third ordinary convolution feature through the third standard normalization layer to obtain a third normalized feature, and activating the third normalized feature through the third activation function layer to obtain an ordinary convolution feature.

[0101] The three depth-separable convolution blocks each include a depth convolution layer, a fourth standard normalization layer and a fourth activation function layer, a second point-by-point convolution layer, a fifth standard normalization layer and a fifth activation function layer;

[0102] The first depth-wise separable convolution block is used to convolve the common convolutional features to obtain the first-level depth features. The steps are as follows:

[0103] Performing a depth convolution operation on the common convolution feature through the depth convolution layer to obtain a first depth convolution feature, performing normalization processing on the first depth convolution feature through the fourth standard normalization layer to obtain a fourth normalized feature, performing activation processing on the fourth normalized feature through the fourth activation function layer to obtain a fourth activation feature, performing a point-by-point convolution operation on the fourth activation feature through the second point-by-point convolution layer to obtain a second depth convolution feature, performing normalization processing on the second depth convolution feature through the fifth standard normalization layer to obtain a fifth normalized feature, performing activation processing on the fifth normalized feature through the fifth activation function layer to obtain a first-level depth feature

[0104] The process of the second depth-wise separable convolution block processing the first-level depth features and the third depth-wise separable convolution block processing the second-level depth features are the same as the processing steps of the first depth-wise separable convolution block.

[0105] In the embodiment of the present invention, the CNN-based lightweight backbone pays more attention to spatial contour information and makes the most of depth information to make up for the deficiency of RGB information.

[0106] Preferably, if Figure 8 As shown, the channel feature extraction is performed on the RGB image by the Swin-Transformer V2 encoder to obtain a multi-scale feature set, including:

[0107] The Swin-Transformer V2 encoder includes a plurality of serially connected Transformer basic blocks connected to a partitioning layer; the RGB image is partitioned by the partitioning layer to obtain a plurality of image blocks, and the plurality of image blocks are linearly mapped to obtain window embedding features;

[0108] The window embedding feature is subjected to a window multi-head self-attention operation through the first Transformer basic block to obtain a first initial window attention weight, the first initial window attention weight is added to the window embedding feature to obtain a first window attention weight, a cross-window multi-head self-attention operation is performed on the first window attention weight to obtain a first initial cross-window attention weight, the first initial cross-window attention weight is added to the first window attention weight to obtain a first cross-window attention weight, a feature extraction is performed on the first cross-window attention weight through a multi-layer perceptron to obtain a first perception feature, the first perception feature is added to the first cross-window attention weight to obtain a first-level RGB feature, and the first is subjected to a second Transformer basic block to extract a feature from the first cross-window attention weight. Perform a window multi-head self-attention operation on the cross-window attention weight to obtain a second initial window attention weight, add the second initial window attention weight to the first cross-window attention weight to obtain a second window attention weight, perform a cross-window multi-head self-attention operation on the second window attention weight to obtain a second initial cross-window attention weight, add the second initial cross-window attention weight to the second window attention weight to obtain a second cross-window attention weight, perform feature extraction on the second cross-window attention weight through a multi-layer perceptron to obtain a second perception feature, add the second perception feature to the second cross-window attention weight to obtain a second-level RGB feature, and so on, until the last Transformer basic block obtains the last-level RGB feature;

[0109] The hierarchical RGB features output by each Transformer basic block constitute a multi-scale feature set.

[0110] Specifically, the RGB image is input into the Swin-Transformer V2 encoder, which includes a partitioning layer and four basic blocks. The partitioning layer is connected to the first basic block, and the remaining basic blocks are connected in sequence.

[0111] The RGB image I is divided into non-overlapping P×P small blocks through the partitioning layer. Each small block is flattened and projected into the feature space through linear embedding to obtain the window embedding feature, which is expressed as:

[0112] X=Linear(Patch(I)),

[0113] Where X is the window embedding feature, Patch(·) divides the RGB image into P×P small blocks, and Linear(·) converts each small block to a higher-dimensional embedding feature space.

[0114] Features are updated layer by layer through hierarchical feature extraction. Each layer of basic blocks includes two core operations: window multi-head self-attention operation and cross-window multi-head self-attention operation. The processing process of the first layer of basic blocks is as follows:

[0115] Perform a window multi-head self-attention operation on the window embedding feature to obtain the first initial window attention weight. Add the first initial window attention weight to the window embedding feature (i.e., perform a patch merging operation) to obtain the first window attention weight, which is expressed as:

[0116] Z l =W-MSA(X l )+X l ,

[0117] Among them, Z l is the window attention weight, W-MSA(·) is the window multi-head self-attention operation, X l is the input feature of the l-th layer basic block.

[0118] Perform a cross-window multi-head self-attention operation on the first window attention weight to obtain the first initial cross-window attention weight, add the first initial cross-window attention weight to the first window attention weight to obtain the first cross-window attention weight, perform feature extraction on the first cross-window attention weight through a multi-layer perceptron to obtain the first perception feature, add the first perception feature to the first cross-window attention weight to obtain the first-level RGB feature Expressed as:

[0119] X l+1 =SW-MSA(Z l )+Z l ,

[0120]

[0121] Among them, SW-MSA(·) is a cross-window multi-head self-attention operation, X l+1 is the cross-window attention weight, is the first-level RGB feature, and MLP(·) is the multi-layer perceptron operation.

[0122] The processing of the remaining three basic blocks is the same as that of the first basic block, and the input of the second basic block is the first-level RGB feature output of the first basic block. The input of the third basic block is the second-level RGB feature output by the second basic block The input of the fourth basic block is the third-level RGB feature output by the third basic block

[0123] By the first RGB feature Second RGB feature Third RGB feature and the fourth RGB feature Composed of multi-scale feature set F rgb for

[0124] It should be understood that a multilayer perceptron (MLP) is a forward-structured artificial neural network whose basic structure includes an input layer, an output layer, and at least one or more hidden layers. Each layer is composed of multiple neurons, and the neurons generate outputs by performing weighted summation on the input values ​​and through an activation function.

[0125] In the embodiment of the present invention, the backbone based on Swin-Transformer V2 extracts more complete details of the RGB image.

[0126] Preferably, if Figure 9 As shown, the depth feature set includes multiple hierarchical depth features, the multi-scale feature set includes multiple hierarchical RGB features, and the multiple hierarchical depth features correspond to the multiple hierarchical RGB features in one-to-one correspondence;

[0127] The hybrid attention module is used to fuse the deep feature set and the multi-scale feature set to obtain multiple different hierarchical image features, including:

[0128] The hybrid attention module includes a Transformer hybrid block, a channel attention block, and a spatial attention block connected in sequence;

[0129] The Transformer mixing block is used to cross-calculate any level depth feature and the corresponding level RGB feature to obtain a first mixed feature and a second mixed feature, and the first mixed feature and the second mixed feature are added to obtain an initial level image feature. The channel information of the initial level image feature is weighted by the channel attention block to obtain a channel attention weight, and the channel attention weight is multiplied element-by-element by the initial level image feature to obtain a channel-level image feature. The spatial information of the channel-level image feature is weighted by the spatial attention block to obtain a spatial attention weight, and the spatial attention weight is multiplied channel-by-channel by the initial level image feature to obtain a spatial-level image feature. The spatial-level image feature is added element-by-element by the initial level image feature to obtain a level image feature. In this process, all level depth features and all level RGB features are processed to obtain multiple different level image features.

[0130] Specifically, the deep feature set F output by the CNN encoder isd for The multi-scale feature set F output by the Swin-Transformer V2 encoder rgb for Input the hybrid attention module and perform cross calculation through the Transformer hybrid block as follows:

[0131]

[0132] in, is the first mixed feature, is the second mixing feature, is the i-th RGB feature, is the i-th deep feature, Transformer(·) is the hybrid block;

[0133] Cross-calculation is performed through the Transformer hybrid block, including: combining information from two different modalities (i.e., deep features and RGB features ) Cross-input the multi-head self-attention mechanism, that is, the query matrix of RGB features, the key matrix and value matrix of deep features are input into the multi-head self-attention mechanism to obtain the RGB feature attention head, and the query matrix of deep features, the key matrix and value matrix of RGB features are input into the multi-head self-attention mechanism to obtain the deep feature attention head, which is expressed as:

[0134] Head rgb =MSA(Q R ,K D ,V D ),

[0135] Head d =MSA(Q D ,K R ,V R ),

[0136] Among them, Head rgb Head is the RGB feature attention head. d is the deep feature attention head, Q R ,K R ,V R They are the query matrix Query, key matrix Key and value matrix Value of RGB features, Q D ,K D ,V D are the query matrix, key matrix, and value matrix of deep features respectively, and MSA(·) is the multi-head self-attention mechanism;

[0137] The first mixed feature is obtained by analyzing the cross-input modal information through the RGB feature attention head The second mixed feature is obtained by analyzing the cross-input modal information through the deep feature attention head

[0138] Add the information output by the Transformer mixing block after preliminary mixing to obtain the initial level image features The initial level image features The weights are calculated by channel attention mechanism (i.e. channel attention block) and spatial attention mechanism (i.e. spatial attention block) respectively to obtain hierarchical image features {F i |i=1,2,3,4}, specifically:

[0139]

[0140] Among them, F i is the hierarchical image feature, CA(·) is the channel attention mechanism, SA(·) is the spatial attention mechanism, ⊕ represents element-by-element addition, represents channel-by-channel multiplication, ⊙ represents element-by-element multiplication. F1 is the low-level image feature (i.e. the first-level depth feature With the first level RGB feature The first-level image features obtained by fusion processing ), F4 is the high-level image feature (i.e. the fourth-level depth feature and the fourth level RGB features The fourth-level image features obtained by fusion processing ).

[0141] It should be understood that the number of features in the deep feature set is the same as the number of features in the multi-scale feature set, so as to fuse the features of two different modalities in a hierarchical order.

[0142] In the embodiment of the present invention, features of two different modalities are cross-fused to obtain fused features corresponding to different levels of RGB information and depth information.

[0143] Preferably, if Figure 10 As shown, the plurality of hierarchical image features include high-level image features and low-level image features;

[0144] Extracting edge features from the plurality of hierarchical image features by the edge constraint module includes:

[0145] The edge constraint module includes a plurality of upsampling blocks;

[0146] The high-level image features are upsampled by the first upsampling block to obtain edge high-level image features, the low-level image features are upsampled by the second upsampling block to obtain edge low-level image features, the edge high-level image features and the edge low-level image features are feature-connected to obtain initial edge features, the initial edge features are upsampled by the third upsampling block to obtain sampled edge features, and the sampled edge features are upsampled by the last upsampling block to obtain edge features.

[0147] Specifically, the edge constraint module includes 4 upsampling blocks, and the upsampling block includes a normal convolution layer and an upsampling layer, specifically: the first upsampling block includes a second 1×1 convolution layer and a first upsampling layer, the second upsampling block includes a third 1×1 convolution layer and a second upsampling layer, the third upsampling block includes a third 3×3 convolution layer and a third upsampling layer, and the fourth upsampling block includes a fourth 1×1 convolution layer and a fourth upsampling layer.

[0148] The high-level image features are convolved by a first common convolution layer, and the high-level image features after convolution are upsampled by a first upsampling layer to obtain edge high-level image features. The low-level image features are convolved by a second common convolution layer, and the low-level image features after convolution are upsampled by a second upsampling layer to obtain edge low-level image features. The edge high-level image features and the edge low-level image features are feature-connected to obtain initial edge features, which can be expressed as:

[0149]

[0150] in, is the initial edge feature, Cat is the feature connection operation, CU1 is the 1×1 convolution layer and upsampling operation, F1 is the low-level image feature (i.e. the first-level image feature ), F4 is the high-level image feature (i.e. the fourth-level image feature ).

[0151] The initial edge features are convolved by the third convolution layer, and the initial edge features after convolution are upsampled by the third upsampling layer to obtain sampled edge features. The sampled edge features are convolved by the fourth convolution layer, and the sampled edge features after convolution are upsampled by the fourth upsampling layer to obtain edge features, which can be expressed as:

[0152]

[0153] Among them, F edge For edge features, CU3 is a 3×3 convolution layer and upsampling operation.

[0154] In an embodiment of the present invention, shallow features are fused with deep features, and the edges of salient objects are used as supervisory information in sequence, so that the output edge information provides edge guidance for the model in the absence of an edge truth map.

[0155] Preferably, if Figure 11 As shown, the progressive decoder is used to decode the plurality of hierarchical image features and the edge features to obtain a salient target image, including:

[0156] The progressive decoder includes a plurality of second normal convolution blocks and a plurality of normal convolution layers;

[0157] Convolution operations are performed on the hierarchical image features one by one through multiple second ordinary convolution blocks respectively, and multiple ordinary convolution hierarchical image features are added element by element to obtain initial salient features. The edge features are activated by an activation function to obtain activated edge features. The activated edge features are multiplied element by element with the initial salient features to obtain initial edge salient features. The initial edge salient features are added element by element with the initial salient features to obtain edge salient features. Ordinary convolution operations are performed on the edge salient features in sequence through multiple ordinary convolution layers to obtain a salient target image.

[0158] Specifically, the plurality of said hierarchical image features {F i |i=1,2,3,4} includes first-level image features (i.e., low-level image features), second-level image features, third-level image features, and fourth-level image features (i.e., high-level image features);

[0159] The progressive decoder includes a 3×3 convolution layer (i.e., the seventh 3×3 normal convolution layer), a 1×1 convolution layer (i.e., the fifth 1×1 normal convolution layer) and a plurality of second normal convolution blocks;

[0160] The fourth-level image feature is convolved through a second common convolution block, and the third-level image feature is convolved through a second common convolution block. The fourth-level image feature after convolution and the third-level image feature after convolution are added element by element to obtain the first initial intermediate feature. The first initial intermediate feature is upsampled to obtain the first intermediate feature, which is expressed as:

[0161] F 3_4 =TB(F3)⊕TB(F4),

[0162] Among them, F 3_4 is the first intermediate feature, F3 is the third-level image feature, F4 is the fourth-level image feature, TB(·) is the second ordinary convolution block (composed of three 3×3 convolution layers), and ⊕ is element-wise addition;

[0163] A convolution operation is performed on the first intermediate feature through a second common convolution block, a convolution operation is performed on the second-level image feature through a second common convolution block, the first intermediate feature after convolution and the second-level image feature after convolution are added element by element to obtain a second initial intermediate feature, and an upsampling operation is performed on the second initial intermediate feature to obtain a second intermediate feature; a convolution operation is performed on the second intermediate feature through a second common convolution block, a convolution operation is performed on the first-level image feature through a second common convolution block, the second intermediate feature after convolution and the first-level image feature after convolution are added element by element to obtain a third initial intermediate feature, and an upsampling operation is performed on the third initial intermediate feature to obtain a third intermediate feature; a convolution operation is performed on the third intermediate feature through a second common convolution block, and an upsampling operation is performed on the third intermediate feature after convolution to obtain an initial salient feature, which can be expressed as:

[0164] S f =δ ↑ (TB(δ ↑ (TB(F i )⊕TB(F i-1 )))),

[0165] Among them, S f is the initial salient feature, TB is the second ordinary convolution block, F i is the i-th level image feature, δ ↑ is the upsampling operation, ⊕ is the element-by-element addition;

[0166] The edge features output by the edge constraint module are input into the progressive decoder to guide the highlighting of edge information in the initial salient features, and the salient features are obtained through a 3×3 convolution layer, which is expressed as:

[0167]

[0168] Among them, S t As a notable feature, S f is the initial salient feature, F edge is the edge feature, Conv3(·) is the convolution operation of 3×3 convolution kernel, Sig(·) is the Sigmoid activation function, ⊕ is element-by-element addition, is element-wise multiplication;

[0169] Through a 1×1 convolution layer as a projection head of salient features, the salient target image S is output o , expressed as:

[0170] S o =Conv1(S t ),

[0171] Among them, So is the salient target image, S t is a significant feature, and Conv1(·) is a convolution operation with a 1×1 convolution kernel.

[0172] The second ordinary convolution block includes three 3×3 ordinary convolution layers connected in series, that is, the fourth 3×3 ordinary convolution layer, the fifth 3×3 ordinary convolution layer, and the sixth 3×3 ordinary convolution layer are sequentially connected to form a second ordinary convolution block; the convolution operation on the first intermediate feature through a second ordinary convolution block and the convolution operation on the second-level image feature through a second ordinary convolution block include:

[0173] The first intermediate feature is subjected to an ordinary convolution operation through the fourth 3×3 ordinary convolution layer to obtain an initial convolution intermediate feature, the initial convolution intermediate feature is subjected to an ordinary convolution operation through the fifth 3×3 ordinary convolution layer to obtain a convolution intermediate feature, and the convolution intermediate feature is subjected to an ordinary convolution operation through the sixth 3×3 ordinary convolution layer to obtain a new convolution intermediate feature (i.e., the fourth-level image feature after convolution).

[0174] The process of performing convolution operations on the remaining second common convolution blocks is the same as the process of performing convolution operations on the second-level image features through one second common convolution block.

[0175] In an embodiment of the present invention, a progressive ladder decoder is used to progressively decode each layer of hierarchical image features, making good use of features at different levels at different stages to improve the performance of the model.

[0176] Preferably, if Figure 10 As shown, the edge constraint module is used to extract the edge of the salient target image to obtain edge supervision features, including:

[0177] The edges of the salient target image obtained in the current training round are extracted using the Sobel function to obtain edge supervision features, which are expressed as:

[0178]

[0179] in, is the edge supervision feature, S o is the salient target image, sobel(·) is the Sobel function used to extract image edges;

[0180] In each round of training, the generation of edge supervision features is guided by the optimized salient target image of each round, that is, edge information is extracted from the salient target image. In this way, the model's ability to predict image edges is improved, resulting in a salient target map with more complete filamentous structures and richer edge details.

[0181] It should be understood that the Sobel function (i.e., the Sobel operator) is a weighted difference of the grayscale values ​​of the four areas of up, down, left, and right of each pixel in the image, which reaches an extreme value at the edge to detect the edge, and is mainly used for edge detection.

[0182] In the embodiment of the present invention, the edge supervision feature is used to guide the edge constraint module to learn edge information during training, and the cross entropy loss and Dice similarity coefficient loss are subsequently used to constrain the training of the edge constraint module.

[0183] Preferably, the loss calculation is performed on the edge feature, the edge supervision feature, the salient target image, and the pseudo label to obtain a total loss (i.e., a mixed loss), including:

[0184] The edge features and the edge supervision features are calculated using an edge cross entropy loss function to obtain an edge cross entropy loss. The edge cross entropy loss function is:

[0185]

[0186] in, is the marginal cross entropy loss, F edge is the edge feature, is the edge supervision feature;

[0187] The pixel values ​​of the edge feature and the edge supervision feature are calculated using a similarity coefficient loss function to obtain a similarity coefficient loss. The similarity coefficient loss function is:

[0188]

[0189] in, is the similarity coefficient loss, p i is the edge feature F edge The i-th pixel value, g i is the edge supervision feature The i-th pixel value of N1 is the total number of pixels of the salient target image;

[0190] The edge cross entropy loss and the similarity coefficient loss are summed to obtain the edge loss. The edge loss summation expression is:

[0191]

[0192] in, is the edge loss, ω is the weighted binary coefficient;

[0193] The target cross entropy loss function is used to calculate the salient target image and the pseudo label to obtain the target cross entropy loss. The target cross entropy loss function is:

[0194]

[0195] in, is the target cross entropy loss, S o is a salient target image, and P is a pseudo label;

[0196] The edge pixel values ​​of the salient target image and the edge pixel values ​​of the pseudo label are calculated using an edge-related loss function to obtain an edge-related loss. The edge-related loss function is:

[0197]

[0198] in, is the edge-related loss, K is the salient target image S o The edge similarity relationship with the pseudo label P, W is the width of the pseudo label, and H is the height of the pseudo label;

[0199] The pixel values ​​of the salient target image and the pixel values ​​of the pseudo label are calculated using a region-related loss function to obtain a region-related loss. The region-related loss function is:

[0200]

[0201] in, is the region-related loss, N2 is the total number of pseudo-labeled pixels;

[0202] The edge-related loss and the region-related loss are summed to obtain a hybrid enhancement loss. The summation expression of the hybrid enhancement loss is:

[0203]

[0204] in, Enhance losses for hybrids;

[0205] The target cross entropy loss and the hybrid enhancement loss are summed to obtain the significance loss, and the summation expression of the significance loss is:

[0206]

[0207] in, Significant loss;

[0208] The pixel value of the salient target image and the pixel value of the pseudo label are calculated using a pixel-level contrast loss function to obtain a pixel-level contrast loss. The pixel-level contrast loss function is:

[0209]

[0210] in, is the pixel-level contrast loss, M j,k is the salient target image S o The same area as the pixel of the pseudo label P, S j,k is the salient target image S o The pixel similarity value with the pseudo label P, j is the jth pixel of the pseudo label P, and k is the salient target image S o The kth pixel of

[0211] The pixel-level contrast loss, the saliency loss, and the edge loss are summed to obtain a total loss. The summation expression of the total loss is:

[0212]

[0213] in, is the total loss, γ is the first learning parameter, set to 0.9, and β is the second learning parameter, set to 0.1.

[0214] In this embodiment of the present invention, the output edge features can effectively address the problem of missing edge information due to sparse labels in weakly supervised models. The pseudo labels are used to guide the salient target images output by the model, thereby improving the model's detection accuracy.

[0215] like Figure 12 As shown, an embodiment of the present invention provides an image saliency detection system based on multi-source weak supervision, including:

[0216] Import unit, used to import RGB images, depth images, text labels and point labels;

[0217] a labeling unit, configured to match the RGB image with the text label to obtain an initial pseudo label, and optimize the initial pseudo label based on the point label to obtain a pseudo label;

[0218] A construction unit for constructing a salient object detection model, wherein the salient object detection model includes a dual branch of a CNN encoder and a Swin-Transformer V2 encoder, and a hybrid attention module, an edge constraint module, and a progressive decoder sequentially connected to the dual branches;

[0219] A prediction unit is configured to perform depth feature extraction on the depth image through the CNN encoder to obtain a depth feature set, perform channel feature extraction on the RGB image through the Swin-Transformer V2 encoder to obtain a multi-scale feature set, fuse the depth feature set and the multi-scale feature set through the hybrid attention module to obtain a plurality of different hierarchical image features, extract edge features from the plurality of hierarchical image features through the edge constraint module, and decode the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a salient target image;

[0220] an optimization unit, configured to perform edge extraction on the salient target image using the edge constraint module to obtain edge supervision features, perform loss calculation on the edge features, the edge supervision features, the salient target image, and the pseudo-label to obtain a total loss, and optimize the salient target detection model using the total loss to obtain an optimized salient target detection model;

[0221] The application unit is configured to detect the RGB image to be detected and the depth image to be detected by using the optimized salient target detection model to obtain a target salient target image.

[0222] The above-mentioned image saliency detection system based on multi-source weak supervision can refer to the implementation content and beneficial effects of the above-mentioned image saliency detection method based on multi-source weak supervision, which will not be repeated here.

[0223] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0224] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for image saliency detection based on multi-source weak supervision, characterized in that: The steps include: Import RGB images, depth images, text labels, and point labels; Matching the RGB image with the text label to obtain an initial pseudo label, and optimizing the initial pseudo label based on the point label to obtain a pseudo label; Constructing a salient object detection model, the salient object detection model including a dual branch of a CNN encoder and a Swin-Transformer V2 encoder, and a hybrid attention module, an edge constraint module, and a progressive decoder sequentially connected to the dual branches; Performing depth feature extraction on the depth image through the CNN encoder to obtain a depth feature set, performing channel feature extraction on the RGB image through the Swin-Transformer V2 encoder to obtain a multi-scale feature set, fusing the depth feature set and the multi-scale feature set through the hybrid attention module to obtain a plurality of different hierarchical image features, extracting edge features from the plurality of hierarchical image features through the edge constraint module, and decoding the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a salient target image; Performing edge extraction on the salient target image using the edge constraint module to obtain edge supervision features, performing loss calculation on the edge features, the edge supervision features, the salient target image, and the pseudo-label to obtain a total loss, and optimizing the salient target detection model using the total loss to obtain an optimized salient target detection model; The optimized salient target detection model is used to detect the RGB image to be detected and the depth image to be detected to obtain a target salient target image.

2. The image saliency detection method according to claim 1, wherein: The matching of the RGB image and the text label to obtain an initial pseudo label includes: Perform feature extraction on the RGB image and the text label using a CLIP model to obtain encoded attention weights, image features, and text features; The probability distribution of the image features and the text features is calculated using the softmax function to obtain a foreground score and a background score. The foreground score and the background score are calculated using a weight calculation expression to obtain multiple pixel point category weights. The weight calculation expression is: in, is the category weight, Z is the total pixel value, y c is the pixel fraction of foreground class c, is the pixel coordinate on the kth channel, s c is the foreground class c score, y c′ is the pixel fraction of background class c′, s c′ is the background class c′ score; Perform weighted summation on the category weights of the plurality of pixel points to obtain an initial category activation map, perform mask extraction on the initial category activation map to obtain a category mask, calculate the encoding attention weight, the initial category activation map, and the category mask using a weight fusion expression to obtain a category activation map, and the weight fusion expression is: Among them, M t is the class activation map, B c is the category mask, W at To encode the attention weight, M c is the initial category activation map, vec(·) is the matrix vectorization, and ⊙ is the element-by-element multiplication; The pixels of the class activation map are filtered according to the set filtering parameters to obtain initial pseudo labels.

3. The image saliency detection method according to claim 1, wherein: The point labels include a foreground target point image and a background target point image; The optimizing the initial pseudo-label based on the point label to obtain the pseudo-label includes: Performing feature extraction on the RGB image using a Sobel algorithm to obtain a rough edge image, and performing pixel filling processing on the rough edge image according to the foreground target point image using a Flood Fill algorithm to obtain a foreground target activation image; The foreground target activation image and the initial pseudo-label are added to obtain a background target image, the background target image is pixel-filled according to the background target point image using the FloodFill algorithm to obtain a background target activation image, and the background target activation image is refined using the conditional random field algorithm to obtain a pseudo-label.

4. The image saliency detection method according to claim 1, wherein: The extracting depth features from the depth image by the CNN encoder to obtain a depth feature set includes: The CNN encoder includes a point-by-point convolution layer, a standard normalization layer, an activation function layer, a first common convolution block and a plurality of depth-separable convolution blocks connected in series, wherein the first common convolution block is connected to the first depth-separable convolution block, the remaining depth-separable convolution blocks are connected in sequence, and the last depth-separable convolution block is connected to the point-by-point convolution layer, the standard normalization layer and the activation function layer in sequence; Performing a convolution operation on the depth image through the first common convolution block to obtain common convolution features; The first depth-wise separable convolution block performs a convolution operation on the common convolution feature to obtain a first-level depth feature, and the second depth-wise separable convolution block performs a convolution operation on the first-level depth feature to obtain a second-level depth feature, and so on, until the last depth-wise separable convolution block performs a convolution operation on the output of the previous depth-wise separable convolution block to obtain a new-level depth feature; Performing a point-by-point convolution operation on the new-level depth feature through the point-by-point convolution layer, normalizing the new-level depth feature after the point-by-point convolution through the standard normalization layer, and activating the normalized new-level depth feature through the activation function layer to obtain the optimal level depth feature; The deep feature set is composed of the output of each depth-separable convolution block and the optimal level deep features.

5. The image saliency detection method according to claim 1, wherein: The step of performing channel feature extraction on the RGB image by the Swin-Transformer V2 encoder to obtain a multi-scale feature set includes: The Swin-Transformer V2 encoder includes a plurality of serially connected Transformer basic blocks connected to a partitioning layer; Performing block processing on the RGB image through the division layer to obtain a plurality of image blocks, and performing linear mapping on the plurality of image blocks to obtain window embedding features; The window embedding feature is subjected to a window multi-head self-attention operation through the first Transformer basic block to obtain a first initial window attention weight, the first initial window attention weight is added to the window embedding feature to obtain a first window attention weight, a cross-window multi-head self-attention operation is performed on the first window attention weight to obtain a first initial cross-window attention weight, the first initial cross-window attention weight is added to the first window attention weight to obtain a first cross-window attention weight, a feature extraction is performed on the first cross-window attention weight through a multi-layer perceptron to obtain a first perception feature, the first perception feature is added to the first cross-window attention weight to obtain a first-level RGB feature, and the first is subjected to a second Transformer basic block to extract a feature from the first cross-window attention weight. Perform a window multi-head self-attention operation on the cross-window attention weight to obtain a second initial window attention weight, add the second initial window attention weight to the first cross-window attention weight to obtain a second window attention weight, perform a cross-window multi-head self-attention operation on the second window attention weight to obtain a second initial cross-window attention weight, add the second initial cross-window attention weight to the second window attention weight to obtain a second cross-window attention weight, perform feature extraction on the second cross-window attention weight through a multi-layer perceptron to obtain a second perception feature, add the second perception feature to the second cross-window attention weight to obtain a second-level RGB feature, and so on, until the last Transformer basic block obtains the last-level RGB feature; The hierarchical RGB features output by each Transformer basic block constitute a multi-scale feature set.

6. The image saliency detection method according to claim 1, wherein: The depth feature set includes multiple hierarchical depth features, the multi-scale feature set includes multiple hierarchical RGB features, and the multiple hierarchical depth features and the multiple hierarchical RGB features are in one-to-one correspondence; The hybrid attention module is used to fuse the deep feature set and the multi-scale feature set to obtain multiple different hierarchical image features, including: The hybrid attention module includes a Transformer hybrid block, a channel attention block, and a spatial attention block connected in sequence; The Transformer mixing block is used to cross-calculate any level depth feature and the corresponding level RGB feature to obtain a first mixed feature and a second mixed feature, and the first mixed feature and the second mixed feature are added to obtain an initial level image feature. The channel information of the initial level image feature is weighted by the channel attention block to obtain a channel attention weight, and the channel attention weight is multiplied element-by-element by the initial level image feature to obtain a channel-level image feature. The spatial information of the channel-level image feature is weighted by the spatial attention block to obtain a spatial attention weight, and the spatial attention weight is multiplied channel-by-channel by the initial level image feature to obtain a spatial-level image feature. The spatial-level image feature is added element-by-element by the initial level image feature to obtain a level image feature. In this process, all level depth features and all level RGB features are processed to obtain multiple different level image features.

7. The image saliency detection method according to claim 1, wherein: The plurality of hierarchical image features include high-level image features and low-level image features; Extracting edge features from the plurality of hierarchical image features by the edge constraint module includes: The edge constraint module includes a plurality of upsampling blocks; The high-level image features are upsampled by the first upsampling block to obtain edge high-level image features, the low-level image features are upsampled by the second upsampling block to obtain edge low-level image features, the edge high-level image features and the edge low-level image features are feature-connected to obtain initial edge features, the initial edge features are upsampled by the third upsampling block to obtain sampled edge features, and the sampled edge features are upsampled by the last upsampling block to obtain edge features.

8. The image saliency detection method according to claim 1, wherein: The step of decoding the plurality of hierarchical image features and the edge features by the progressive decoder to obtain a salient target image comprises: The progressive decoder includes a plurality of second normal convolution blocks and a plurality of normal convolution layers; Convolution operations are performed on the hierarchical image features one by one through multiple second ordinary convolution blocks respectively, and multiple ordinary convolution hierarchical image features are added element by element to obtain initial salient features. The edge features are activated by an activation function to obtain activated edge features. The activated edge features are multiplied element by element with the initial salient features to obtain initial edge salient features. The initial edge salient features are added element by element with the initial salient features to obtain edge salient features. Ordinary convolution operations are performed on the edge salient features in sequence through multiple ordinary convolution layers to obtain a salient target image.

9. The image saliency detection method according to claim 1, wherein: The loss calculation of the edge feature, the edge supervision feature, the salient target image, and the pseudo label to obtain a total loss includes: The edge features and the edge supervision features are calculated using an edge cross entropy loss function to obtain an edge cross entropy loss. The edge cross entropy loss function is: in, is the marginal cross entropy loss, F edge is the edge feature, is the edge supervision feature; The pixel values ​​of the edge feature and the edge supervision feature are calculated using a similarity coefficient loss function to obtain a similarity coefficient loss. The similarity coefficient loss function is: in, is the similarity coefficient loss, p i is the i-th pixel value of the edge feature, g i is the i-th pixel value of the edge supervision feature, and N1 is the total number of pixels of the salient target image; Summing the edge cross entropy loss and the similarity coefficient loss to obtain the edge loss; The target cross entropy loss function is used to calculate the salient target image and the pseudo label to obtain the target cross entropy loss. The target cross entropy loss function is: in, is the target cross entropy loss, S o is a salient target image, and P is a pseudo label; The edge pixel values ​​of the salient target image and the edge pixel values ​​of the pseudo label are calculated using an edge-related loss function to obtain an edge-related loss. The edge-related loss function is: in, is the edge correlation loss, K is the edge similarity relationship, W is the width of the pseudo label, and H is the height of the pseudo label; The pixel values ​​of the salient target image and the pixel values ​​of the pseudo label are calculated using a region-related loss function to obtain a region-related loss. The region-related loss function is: in, is the region-related loss, N2 is the total number of pseudo-labeled pixels; The edge-related loss and the region-related loss are summed to obtain a hybrid enhancement loss, and the target cross entropy loss and the hybrid enhancement loss are summed to obtain a saliency loss; The pixel value of the salient target image and the pixel value of the pseudo label are calculated using a pixel-level contrast loss function to obtain a pixel-level contrast loss. The pixel-level contrast loss function is: in, is the pixel-level contrast loss, M j,k For the same pixel area, S j,k is the pixel similarity value, j is the jth pixel of the pseudo label, and k is the kth pixel of the salient target image; The pixel-level contrast loss, the saliency loss, and the edge loss are summed to obtain a total loss.

10. An image saliency detection system based on multi-source weak supervision, characterized in that: include: Import unit, used to import RGB images, depth images, text labels and point labels; a labeling unit, configured to match the RGB image with the text label to obtain an initial pseudo label, and optimize the initial pseudo label based on the point label to obtain a pseudo label; A construction unit for constructing a salient object detection model, wherein the salient object detection model includes a dual branch of a CNN encoder and a Swin-Transformer V2 encoder, and a hybrid attention module, an edge constraint module, and a progressive decoder sequentially connected to the dual branches; A prediction unit is configured to perform depth feature extraction on the depth image through the CNN encoder to obtain a depth feature set, perform channel feature extraction on the RGB image through the Swin-Transformer V2 encoder to obtain a multi-scale feature set, fuse the depth feature set and the multi-scale feature set through the hybrid attention module to obtain a plurality of different hierarchical image features, extract edge features from the plurality of hierarchical image features through the edge constraint module, and decode the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a salient target image; an optimization unit, configured to perform edge extraction on the salient target image using the edge constraint module to obtain edge supervision features, perform loss calculation on the edge features, the edge supervision features, the salient target image, and the pseudo-label to obtain a total loss, and optimize the salient target detection model using the total loss to obtain an optimized salient target detection model; The application unit is configured to detect the RGB image to be detected and the depth image to be detected by using the optimized salient target detection model to obtain a target salient target image.

Images