Image saliency detection method and system based on multi-source weak supervision

Abstract

The invention provides an image saliency detection method and system based on multi-source weak supervision, and relates to the technical field of image processing. The method comprises the following steps: marking an RGB image based on a text label and a point label to generate a pseudo label, performing feature extraction on the RGB image and a depth image by using different encoders in a constructed saliency target detection model, and performing fusion processing on the extracted features according to levels to obtain a plurality of different level image features, and generating edge features according to the plurality of hierarchical image features, and decoding the plurality of hierarchical image features in combination with the edge features to obtain a saliency target image. By combining the advantages of various weak labels, the supplementary effect of depth information on RGB information is fully played, so that a significance target detected by a significance target detection model highlights a filament-shaped edge structure.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of image processing, and in particular to an image saliency detection method and system based on multi-source weak supervision. Background Art

[0002] Salient object detection (SOD) aims to locate and segment the objects that attract human attention in a scene. Existing salient object detection technologies have made significant progress in using multimodal information to highlight objects based on appearance information. Among them, the image salient object detection task with depth information (RGB-D SOD) has become a mainstream technology. With the development of deep learning, RGB-D SOD based on full supervision can achieve high-precision object detection, but it is heavily dependent on pixel-level annotations. The acquisition process of pixel-level annotations is very cumbersome and the annotation cost is expensive. Therefore, salient object detection methods using weak supervision training have emerged.

[0003] In the current mainstream RGB-D SOD research based on weak supervision, using single-source weak annotations as supervision will result in weak richness of pixel training samples, poor structural integrity of salient objects, and weak prominence of salient targets. It can be seen that a single weak annotation is difficult to complete the SOD task. Using multiple weak annotations as supervision also has the problem of conflicting annotation characteristics, which ultimately leads to the degradation of the advantages of each annotation, resulting in the output results lacking significant prominence in both details and overall performance. Since the input information of the RGB-D SOD task is RGB images and depth images, most weakly supervised RGB-D SOD tasks simply put the two into the same backbone network for processing, which will inevitably lead to the final result that the salient target and the things around the salient target are regarded as salient targets. Therefore, it is urgent to solve the problems of poor structural integrity and weak prominence of salient targets caused by weak labels. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an image saliency detection method and system based on multi-source weak supervision in view of the deficiencies in the prior art.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] A method for image saliency detection based on multi-source weak supervision includes the following steps:

[0007] Import RGB images, depth images, text labels, and point labels;

[0008] Matching the RGB image with the text label to obtain an initial pseudo label, and optimizing the initial pseudo label based on the point label to obtain a pseudo label;

[0009] Constructing a salient object detection model, wherein the salient object detection model includes a dual branch of a CNN encoder and a Swin-Transformer V2 encoder, and a hybrid attention module, an edge constraint module, and a progressive decoder respectively connected to the dual branches in sequence;

[0010] Performing deep feature extraction on the depth image through the CNN encoder to obtain a deep feature set, performing channel feature extraction on the RGB image through the Swin-Transformer V2 encoder to obtain a multi-scale feature set, performing fusion processing on the deep feature set and the multi-scale feature set through the hybrid attention module to obtain a plurality of different hierarchical image features, extracting edge features from the plurality of hierarchical image features through the edge constraint module, and respectively decoding the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a salient target image;

[0011] Performing edge extraction on the salient target image by using the edge constraint module to obtain edge supervision features, performing loss calculation on the edge features, the edge supervision features, the salient target image and the pseudo-label to obtain a total loss, and optimizing the salient target detection model by using the total loss to obtain an optimized salient target detection model;

[0012] The optimized salient target detection model is used to detect the RGB image to be detected and the depth image to be detected to obtain a target salient target image.

[0013] Another technical solution of the present invention to solve the above technical problems is as follows:

[0014] An image saliency detection system based on multi-source weak supervision, comprising:

[0015] Import unit, used to import RGB images, depth images, text labels and point labels;

[0016] A labeling unit, used for matching the RGB image with the text label to obtain an initial pseudo label, and optimizing the initial pseudo label based on the point label to obtain a pseudo label;

[0017] A construction unit, used to construct a salient object detection model, wherein the salient object detection model includes a dual branch of a CNN encoder and a Swin-Transformer V2 encoder, and a hybrid attention module, an edge constraint module, and a progressive decoder respectively connected to the dual branches in sequence;

[0018] A prediction unit, configured to extract deep features from the depth image through the CNN encoder to obtain a deep feature set, extract channel features from the RGB image through the Swin-Transformer V2 encoder to obtain a multi-scale feature set, fuse the deep feature set and the multi-scale feature set through the hybrid attention module to obtain a plurality of different hierarchical image features, extract edge features from the plurality of hierarchical image features through the edge constraint module, and respectively decode the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a salient target image;

[0019] an optimization unit, configured to extract edges of the salient target image through the edge constraint module to obtain edge supervision features, perform loss calculation on the edge features, the edge supervision features, the salient target image and the pseudo-label to obtain a total loss, and optimize the salient target detection model through the total loss to obtain an optimized salient target detection model;

[0020] The application unit is used to detect the RGB image to be detected and the depth image to be detected by using the optimized salient target detection model to obtain a target salient target image.

[0021] The beneficial effects of the present invention are as follows: pseudo labels are generated by combining text information and image information from multiple sources, which makes up for the inherent defects of weak labels and enables the model to learn the structural information of significant objects from annotations. Different types of images are extracted separately through two different backbone networks, the CNN encoder and the Swin-Transformer V2 encoder, thereby avoiding the feature information containing too much interference noise and being able to extract the required features in a targeted manner. The edge features of the image are extracted by the edge constraint module, so that significant features can be identified more clearly. The features of each level are decoded by a progressive decoder to identify all the features that show significance from the features of each level, so as to work together to exert the feature decoding performance of the model. A variety of loss functions are mixed to guide the training of the model so that the model can detect more accurate results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of an image saliency detection method based on multi-source weak supervision provided by an embodiment of the present invention;

[0023] Figure 2 An RGB image provided by an embodiment of the present invention;

[0024] Figure 3 A depth image provided by an embodiment of the present invention;

[0025] Figure 4A salient target image provided by an embodiment of the present invention;

[0026] Figure 5 A structural diagram of a salient object detection model provided by an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of a pseudo-label generation process provided by an embodiment of the present invention;

[0028] Figure 7 A structural diagram of a CNN encoder provided in an embodiment of the present invention;

[0029] Figure 8 A structural diagram of the Swin-Transformer V2 encoder provided in an embodiment of the present invention;

[0030] Fig. 9 A structural diagram of a hybrid attention module provided in an embodiment of the present invention;

[0031] Fig.10 A structural diagram of an edge constraint module provided in an embodiment of the present invention;

[0032] Fig.11 A structural diagram of a progressive decoder provided by an embodiment of the present invention;

[0033] Fig.12 A unit block diagram of an image saliency detection system based on multi-source weak supervision provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0035] Salient object detection models usually require a large number of pixel-level labeled samples for training. However, due to the high cost of strong supervision sources, saliency detection models based on weak supervision sources (such as classification labels, summary labels, etc.) have become a research hotspot. By integrating information from multiple weak supervision sources, the reliance on high-quality labeled data can be reduced, thereby reducing the labeling cost and improving the generalization ability of the model. Multi-source weakly supervised learning uses partial, imprecise or other inaccurate information for training, which is easier to obtain than manually labeled data, thereby reducing the labeling cost and being applied in scenarios where labeled data is difficult to obtain or the labeling cost is high.

[0036] In the weakly supervised RGB-D SOD task, using single-source weak annotations as supervision will lead to several serious problems, namely, the weak richness of pixel training samples, poor structural integrity of salient objects, and weak prominence of salient targets. Due to the weak richness of pixel training samples, single-source weak annotations cannot provide enough important pixel samples for robust saliency perception learning. The poor structural integrity of salient objects does not allow the RGB-D SOD model to learn the structural information of salient objects directly from the annotations. The weak prominence of salient targets may highlight background targets, which goes against the original intention of the SOD task. Using multi-source annotations as supervision will lead to the problem of conflicting annotation features, which ultimately leads to the degradation of the advantages of each annotation, and the results of the model output lack outstanding performance in both details and integrity. Since the input information of the RGB-D SOD task is RGB images and depth images, which are essentially different, most of the current weakly supervised RGB-D SOD tasks simply put the RGB images and depth images into the same backbone, which will inevitably lead to the extraction of a lot of noise information, making it difficult to fully express the features of the depth image, and even weakening the detailed information of the RGB image features, resulting in the final result that the salient target and the things around the salient target are regarded as salient targets, which also violates the original intention of the SOD task.

[0037] like Figure 1-Figure 5 As shown, an image saliency detection method based on multi-source weak supervision provided by an embodiment of the present invention includes the following steps:

[0038] Import RGB images, depth images, text labels, and point labels;

[0039] Matching the RGB image with the text label to obtain an initial pseudo label, and optimizing the initial pseudo label based on the point label to obtain a pseudo label;

[0040] Constructing a salient object detection model, wherein the salient object detection model includes a dual branch of a CNN encoder and a Swin-Transformer V2 encoder, and a hybrid attention module, an edge constraint module, and a progressive decoder respectively connected to the dual branches in sequence;

[0041] Performing deep feature extraction on the depth image through the CNN encoder to obtain a deep feature set, performing channel feature extraction on the RGB image through the Swin-Transformer V2 encoder to obtain a multi-scale feature set, performing fusion processing on the deep feature set and the multi-scale feature set through the hybrid attention module to obtain a plurality of different hierarchical image features, extracting edge features from the plurality of hierarchical image features through the edge constraint module, and respectively decoding the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a salient target image;

[0042] Performing edge extraction on the salient target image by using the edge constraint module to obtain edge supervision features, performing loss calculation on the edge features, the edge supervision features, the salient target image and the pseudo-label to obtain a total loss, and optimizing the salient target detection model by using the total loss to obtain an optimized salient target detection model;

[0043] The optimized salient target detection model is used to detect the RGB image to be detected and the depth image to be detected to obtain a target salient target image.

[0044] It should be understood that RGB-D images include RGB images and depth images. RGB images are RGB three-channel color images, and Depth images (i.e., depth images) contain image channels that contain information related to the surface distance of the viewpoint scene object. The channel itself is similar to a grayscale image, and each pixel value is the actual distance measured by the sensor to the object.

[0045] In the embodiment of the present invention, in order to solve the problems of poor structural integrity and weak prominence of salient targets caused by weak labels in weak supervision, the present invention combines text labels and point labels to complete the construction of weak supervision pseudo labels, uses asymmetric dual-branch backbone extraction features according to the different characteristics of input images, and uses low-level features and high-level features to generate edge features. At the same time, the edge of the salient target image predicted by the model is used as a guide, and the model is optimized using a loss function to improve the quality of the salient target map and edge map output by the model, thereby improving the model detection accuracy.

[0046] Preferably, if Figure 6 As shown, the RGB image and the text label are matched to obtain an initial pseudo label, including:

[0047] Perform feature extraction on the RGB image and the text label through the CLIP model to obtain encoding attention weights, image features, and text features;

[0048] The image features and the text features are mapped by the Grad-CAM algorithm to obtain a category activation map, including: calculating the probability distribution of the image features and the text features by the softmax function to obtain a foreground class score and a background class score, and calculating the foreground class score and the background class score by a weight calculation expression to obtain a plurality of pixel point category weights, wherein the weight calculation expression is:

[0049]

[0050] in, is the category weight of the foreground class c in the kth channel in the last feature layer A of the CLIP model, Z is the total pixel value, and y c is the foreground pixel score (i.e., the score corresponding to the pixel points of foreground class c in the forward propagation of the CLIP model), is the pixel coordinate (i, j) on the kth channel in the last feature layer A of the CLIP model, s c is the foreground class score, y c′ is the background pixel score (i.e., the score corresponding to the background pixel in the forward propagation of the CLIP model), s c′ is the background score, c is the foreground score, c′ is the total category (including foreground and background. Since c′≠c, c′ is the background score);

[0051] is the gradient information of the foreground class back-propagated on the last feature layer A of the CLIP model, The gradient information of the background class back-propagated on the last feature layer A of the CLIP model;

[0052] The weighted summation of the multiple pixel category weights is performed to obtain an initial category activation map, the mask extraction is performed on the initial category activation map to obtain a category mask, and the encoding attention weight, the initial category activation map and the category mask are calculated by a weight fusion expression to obtain a category activation map, and the weight fusion expression is:

[0053]

[0054] Among them, M t is the category activation map, B c is the class mask, W at To encode the attention weight, M c is the initial category activation map, vec(·) is the matrix vectorization, and ⊙ is the element-by-element multiplication;

[0055] The pixels of the class activation map are filtered according to the set filtering parameters to obtain initial pseudo labels.

[0056] Specifically, the expression of the softmax function is:

[0057]

[0058] Among them, s c is the foreground score, exp(·) is the exponential function, y c is the foreground pixel fraction, y c′ is the total category pixel score, c′ is the total category, c′={1,...,c,...,N}, N is the number of categories.

[0059] The weighted sum of the multiple pixel point category weights is performed using a weighted expression, and the weighted expression is:

[0060]

[0061] Among them, ReLU(·) is the activation process.

[0062] The processing process of the weight fusion expression is as follows: the initial category activation map M of the foreground target class c Thresholding is performed, and the initial category activation map M c Find the connected areas on the graph and get the minimum rectangular bounding box covering these connected areas to get the category mask B c , and then compared with the encoded attention weight W output by the multi-head attention layer of the image encoder in the CLIP model at Combined, encoding attention weight W at For pixel-by-pixel activation of the initial category map M c Make optimizations.

[0063] Since the category activation map is essentially the confidence of each pixel in the image about the category, a hyperparameter λ (i.e., filtering parameter) is set to ignore pixels with too low confidence and obtain the initial pseudo label, which is expressed as:

[0064]

[0065] Among them, U is the initial pseudo label, (i, j) is the category activation map M t The coordinates of the pixel in .

[0066] It should be understood that the image-text multimodal model (i.e., CLIP model) learns the matching relationship between images and texts by comparison, so that relevant text features and image features are adjacent in space, while irrelevant ones are far away, capturing the deep semantic connection between images and texts and achieving cross-modal understanding. In the present invention, the CLIP model is a pre-trained large network model.

[0067] The Grad-CAM algorithm is used to perform gradient-weighted class activation mapping. The importance of the feature map is determined by calculating the gradient of the feature map with respect to the CLIP model output, that is, classification activation mapping is performed on the feature map. The specific steps include: obtaining the category probability of the model output from the forward propagation of the CLIP model (that is, the CLIP model performs the matching process); calculating the gradient of the output category probability with respect to the feature map based on the target category (that is, the gradient of back propagation); and weightedly superimposing each gradient with the corresponding feature map to obtain a Grad-CAM image (that is, a category activation map).

[0068] In an embodiment of the present invention, text labels and RGB images are sent to the CLIP model to extract text features and image features, and the Grad-CAM technology is used to combine the two to output a class activation map, and the gradient information is back-propagated back to the CLIP model. The attention weight map output by the multi-head attention mechanism in the image encoder of the CLIP model is combined with the class mask extracted from the class activation map to generate a class-aware attention weight map, which is used to improve the class activation map into an initial pseudo-label pixel by pixel.

[0069] Preferably, if Figure 6 As shown, the point labels include a foreground target point image and a background target point image;

[0070] The step of optimizing the initial pseudo-label based on the point label to obtain the pseudo-label includes:

[0071] Performing feature extraction on the RGB image by using the Sobel algorithm to obtain a rough edge image, and performing pixel filling processing on the rough edge image according to the foreground target point image by using the FloodFill algorithm to obtain a foreground target activation image;

[0072] The foreground target activation image and the initial pseudo-label are added to obtain a background target image, and the background target image is pixel-filled according to the background target point image using the FloodFill algorithm to obtain a background target activation image, and the background target activation image is refined using a conditional random field algorithm to obtain a pseudo-label.

[0073] Specifically, the process of obtaining the foreground target activation image is expressed as:

[0074] P f =F(S,E),

[0075] Among them, P f is the foreground target activation image, S is the foreground target point image, E is the rough edge image, and F(·) is the Flood Fill algorithm.

[0076] The process of obtaining pseudo labels based on the foreground target activation image is expressed as:

[0077] P = CRF(F(SS b ,P f +U)),

[0078] Among them, P is a pseudo label, S b is the background target point image, U is the initial pseudo label, CRF(·) is the conditional random field algorithm, F(·) is the Flood Fill algorithm, SS b Indicates the subtraction of the foreground target point image and the background target point image.

[0079] It should be understood that the Flood Fill algorithm is a basic algorithm for image processing, which is used to fill connected areas. Usually starting from a set pixel point, fill along the adjacent pixels of the set pixel point until a boundary or other specified conditions are encountered. The Flood Fill algorithm is used to expand the entire image, so that a complete salient target map is obtained, while non-salient targets are also suppressed to obtain refined pseudo-labels. The CRF conditional random field algorithm (CRF) is used to refine the segmentation results of the pseudo-labels.

[0080] In an embodiment of the present invention, an image is annotated with point labels, and a Flood Fill algorithm is used to activate the point annotation area. Then, the foreground activation area is used to supplement the deficiency of the target in the initial pseudo-label, and the background activation area is used to suppress the redundant noise in the initial pseudo-label, so as to obtain a pseudo-label close to the true value image, which is used to guide the training of the salient target detection model.

[0081] High-quality pseudo labels generated by text labels and point labels are used as supervisory signals to guide the training of the salient object detection model, thereby enhancing the model's ability to characterize the filamentous structure of salient objects and achieving accurate detection of salient objects in RGB-D images (RGB images and depth images).

[0082] Preferably, if Figure 7 As shown, the CNN encoder is used to extract the depth features of the depth image to obtain a depth feature set, including:

[0083] The CNN encoder includes a point-by-point convolution layer, a standard normalization layer, an activation function layer, a first common convolution block and a plurality of depth-separable convolution blocks connected in series, wherein the first common convolution block is connected to the first depth-separable convolution block, the remaining depth-separable convolution blocks are connected in sequence, and the last depth-separable convolution block is connected to the point-by-point convolution layer, the standard normalization layer and the activation function layer in sequence;

[0084] Performing a convolution operation on the depth image through the first common convolution block to obtain common convolution features;

[0085] The first depth-separable convolution block performs a convolution operation on the common convolution feature to obtain a first-level depth feature, and the second depth-separable convolution block performs a convolution operation on the first depth feature to obtain a second-level depth feature, and so on, until the last depth-separable convolution block performs a convolution operation on the output of the previous depth-separable convolution block to obtain a new-level depth feature;

[0086] Performing point-by-point convolution operation on the new-level depth feature through the point-by-point convolution layer, normalizing the new-level depth feature after point-by-point convolution through the standard normalization layer, and activating the normalized new-level depth feature through the activation function layer to obtain the optimal level depth feature;

[0087] The output of each depth-separable convolution block and the optimal level deep features form a deep feature set (that is, the deep feature set is composed of multiple levels of deep features).

[0088] Specifically, in the process of extracting deep information features, the CNN-base encoder (i.e., CNN encoder) first uses a backbone consisting of a 1×1 convolutional layer (denoted as Conv 1) and two 3×3 convolutional layers (denoted as Conv 3) to extract deep features for the input deep image D. These layers (1×1 convolutional layer and two 3×3 convolutional layers) are equipped with standard normalization (BN) and ReLU activation functions. After processing, ordinary convolution features are obtained. It is expressed as:

[0089]

[0090] Then, the learned deep features are fed into three stages (i.e., depthwise separable convolution blocks) in sequence. For each stage, the first depthwise separable convolution block performs normal convolution on the features by 3×3 depthwise (DW) convolution and 1×1 pointwise (PW) convolution with BN and ReLU activation functions. After processing, the first level of deep features is obtained The second depth-wise separable convolutional block performs deep convolution on the first-level deep features. After processing, the second-level deep features are obtained The third depth-separable convolutional block performs deep convolution on the second-level deep features. After processing, the third-level deep features are obtained (i.e., new level depth feature), expressed as:

[0091]

[0092] The third-level deep features are processed by a layer of 1×1 point (PW) convolution and BN and ReLU. Perform feature extraction to obtain the fourth-level deep features (i.e., optimal level depth feature), expressed as:

[0093]

[0094] Among them, ReLU is the activation function, BN(·) is the standard normalization process, Conv2d i (·) is a 2D image convolution operation, DWConv3 is a 3×3 depth convolution operation, and PWConv1(·) is a 1×1 point-by-point convolution operation.

[0095] The first-level deep features extracted by outputting multiple depth-separable convolutional blocks Second level deep features The third level deep features And the fourth-level deep features extracted after point-by-point convolution operation Composed deep feature set F d for

[0096] Specifically, the CNN encoder includes a first point-by-point convolution layer, a standard normalization layer, an activation function layer, a first ordinary convolution block and three depth-separable convolution blocks; the first ordinary convolution block includes a first 1×1 ordinary convolution layer, a first standard normalization layer and a first activation function layer, a first 3×3 ordinary convolution layer, a second standard normalization layer and a second activation function layer, a second 3×3 ordinary convolution layer, a third standard normalization layer and a third activation function layer connected in sequence.

[0097] The step of performing a convolution operation on the depth image through the first common convolution block to obtain common convolution features includes:

[0098] Performing a normal convolution operation on the depth image through the first 1×1 normal convolution layer to obtain a first normal convolution feature, performing normalization processing on the first normal convolution feature through the first standard normalization layer to obtain a first normalized feature, and performing activation processing on the first normalized feature through the first activation function layer to obtain a first activated feature;

[0099] Performing a normal convolution operation on the first activation feature through the first 3×3 normal convolution layer to obtain a second normal convolution feature, performing normalization processing on the second normal convolution feature through the second standard normalization layer to obtain a second normalized feature, and performing activation processing on the second normalized feature through the second activation function layer to obtain a second activation feature;

[0100] The second activation feature is subjected to an ordinary convolution operation through the second 3×3 ordinary convolution layer to obtain a third ordinary convolution feature, the third ordinary convolution feature is normalized through the third standard normalization layer to obtain a third normalized feature, and the third normalized feature is activated through the third activation function layer to obtain an ordinary convolution feature.

[0101] The three depth-separable convolution blocks each include a depth convolution layer, a fourth standard normalization layer and a fourth activation function layer, a second point-by-point convolution layer, a fifth standard normalization layer and a fifth activation function layer;

[0102] The first depth-separable convolution block is used to convolve the common convolution feature to obtain the first-level depth feature. The steps are as follows:

[0103] The common convolution feature is subjected to a deep convolution operation through the deep convolution layer to obtain a first deep convolution feature, the first deep convolution feature is subjected to normalization processing through the fourth standard normalization layer to obtain a fourth normalized feature, the fourth normalized feature is subjected to activation processing through the fourth activation function layer to obtain a fourth activated feature, the fourth activated feature is subjected to a point-by-point convolution operation through the second point-by-point convolution layer to obtain a second deep convolution feature, the second deep convolution feature is subjected to normalization processing through the fifth standard normalization layer to obtain a fifth normalized feature, the fifth normalized feature is subjected to activation processing through the fifth activation function layer to obtain a first-level deep feature

[0104] The process of processing the first-level deep features by the second depth-wise separable convolution block and the process of processing the second-level deep features by the third depth-wise separable convolution block are the same as the processing steps of the first depth-wise separable convolution block.

[0105] In the embodiment of the present invention, the CNN-based lightweight backbone pays more attention to spatial contour information and makes the most of depth information to make up for the deficiency of RGB information.

[0106] Preferably, if Figure 8 As shown, the channel feature extraction is performed on the RGB image by the Swin-Transformer V2 encoder to obtain a multi-scale feature set, including:

[0107] The Swin-Transformer V2 encoder includes a plurality of serially connected Transformer basic blocks connected to a partitioning layer; the RGB image is partitioned by the partitioning layer to obtain a plurality of image blocks, and a plurality of image blocks are linearly mapped to obtain a window embedding feature;

[0108] The first Transformer basic block performs a window multi-head self-attention operation on the window embedding feature to obtain a first initial window attention weight, adds the first initial window attention weight to the window embedding feature to obtain a first window attention weight, performs a cross-window multi-head self-attention operation on the first window attention weight to obtain a first initial cross-window attention weight, adds the first initial cross-window attention weight to the first window attention weight to obtain a first cross-window attention weight, performs feature extraction on the first cross-window attention weight through a multi-layer perceptron to obtain a first perceptual feature, adds the first perceptual feature to the first cross-window attention weight to obtain a first-level RGB feature, and performs a multi-layer perceptron on the first cross-window attention weight to obtain a first-level RGB feature. Perform a window multi-head self-attention operation on the cross-window attention weight to obtain a second initial window attention weight, add the second initial window attention weight to the first cross-window attention weight to obtain a second window attention weight, perform a cross-window multi-head self-attention operation on the second window attention weight to obtain a second initial cross-window attention weight, add the second initial cross-window attention weight to the second window attention weight to obtain a second cross-window attention weight, perform feature extraction on the second cross-window attention weight through a multi-layer perceptron to obtain a second perceptual feature, add the second perceptual feature to the second cross-window attention weight to obtain a second-level RGB feature, and so on, until the last Transformer basic block obtains the last-level RGB feature;

[0109] The hierarchical RGB features output by each Transformer basic block constitute a multi-scale feature set.

[0110] Specifically, the RGB image is input into the Swin-Transformer V2 encoder, which includes a partition layer and 4 basic blocks. The partition layer is connected to the first basic block, and the remaining basic blocks are connected in sequence.

[0111] The RGB image I is divided into non-overlapping P×P small blocks through the partitioning layer. Each small block is flattened and projected into the feature space through linear embedding to obtain the window embedding feature, which is expressed as:

[0112] X = Linear(Patch(I)),

[0113] Among them, X is the window embedding feature, Patch(·) is to divide the RGB image into P×P small blocks, and Linear(·) is to convert each small block into a higher-dimensional embedding feature space.

[0114] The features are updated layer by layer through hierarchical feature extraction. Each layer of basic blocks includes two core operations: window multi-head self-attention operation and cross-window multi-head self-attention operation. The processing process of the first layer of basic blocks is as follows:

[0115] The window multi-head self-attention operation is performed on the window embedding feature to obtain the first initial window attention weight. The first initial window attention weight is added to the window embedding feature (i.e., a patch merging operation is performed) to obtain the first window attention weight, which is expressed as:

[0116] Z l =W-MSA(X l )+X l ,

[0117] Among them, Z l is the window attention weight, W-MSA(·) is the window multi-head self-attention operation, X l is the input feature of the l-th layer basic block.

[0118] Perform a cross-window multi-head self-attention operation on the first window attention weight to obtain the first initial cross-window attention weight, add the first initial cross-window attention weight to the first window attention weight to obtain the first cross-window attention weight, perform feature extraction on the first cross-window attention weight through a multi-layer perceptron to obtain the first perceptual feature, add the first perceptual feature to the first cross-window attention weight to obtain the first-level RGB feature It is expressed as:

[0119] X l+1 =SW-MSA(Z l )+Z l ,

[0120]

[0121] Among them, SW-MSA(·) is a cross-window multi-head self-attention operation, X l+1 is the cross-window attention weight, is the first-level RGB feature, and MLP(·) is a multi-layer perceptron operation.

[0122] The processing of the remaining three basic blocks is the same as that of the first basic block, and the input of the second basic block is the first-level RGB feature output of the first basic block. The input of the third basic block is the second-level RGB feature output by the second basic block The input of the fourth basic block is the third-level RGB feature output by the third basic block

[0123] By the first RGB feature Second RGB feature Third RGB feature and the fourth RGB feature Composed of multi-scale feature set F rgb for

[0124] It should be understood that a multilayer perceptron (MLP) is a forward-structured artificial neural network whose basic structure includes an input layer, an output layer, and at least one or more hidden layers. Each layer is composed of multiple neurons. The neurons perform weighted summation of input values ​​and generate outputs through activation functions.

[0125] In the embodiment of the present invention, the backbone based on Swin-Transformer V2 extracts more complete details of the RGB image.

[0126] Preferably, if Fig. 9 As shown, the depth feature set includes multiple levels of depth features, the multi-scale feature set includes multiple levels of RGB features, and the multiple levels of depth features correspond to the multiple levels of RGB features in a one-to-one manner;

[0127] The hybrid attention module is used to fuse the deep feature set and the multi-scale feature set to obtain multiple different levels of image features, including:

[0128] The hybrid attention module includes a Transformer hybrid block, a channel attention block, and a spatial attention block connected in sequence;

[0129] The Transformer mixing block is used to cross-calculate any level of depth feature and the corresponding level of RGB feature to obtain a first mixed feature and a second mixed feature, the first mixed feature and the second mixed feature are added to obtain an initial level image feature, the channel information of the initial level image feature is weighted by the channel attention block to obtain a channel attention weight, the channel attention weight is multiplied element-by-element by the initial level image feature to obtain a channel-level image feature, the spatial information of the channel-level image feature is weighted by the spatial attention block to obtain a spatial attention weight, the spatial attention weight is multiplied channel-by-channel by the initial level image feature to obtain a spatial level image feature, the spatial level image feature is added element-by-element by the initial level image feature to obtain a level image feature, and in this process, all levels of depth features and all levels of RGB features are processed to obtain multiple different levels of image features.

[0130] Specifically, the deep feature set F output by the CNN encoder isd for The multi-scale feature set F output by the Swin-Transformer V2 encoder rgb for Input the hybrid attention module and perform cross calculation through the Transformer hybrid block as follows:

[0131]

[0132] in, is the first mixed feature, is the second mixed feature, is the i-th RGB feature, is the i-th deep feature, Transformer(·) is the hybrid block;

[0133] The Transformer hybrid block performs cross-calculations, including: combining information from two different modalities (i.e., deep features) and RGB features ) cross-input into the multi-head self-attention mechanism, that is, the query matrix of RGB features, the key matrix and value matrix of deep features are input into the multi-head self-attention mechanism to obtain the RGB feature attention head, and the query matrix of deep features, the key matrix and value matrix of RGB features are input into the multi-head self-attention mechanism to obtain the deep feature attention head, which is expressed as:

[0134] Head rgb =MSA(Q R ,K D ,V D ),

[0135] Head d =MSA(Q D ,K R ,V R ),

[0136] Among them, Head rgb Head is the RGB feature attention head. d is the deep feature attention head, Q R ,K R ,V R They are the query matrix Query, key matrix Key and value matrix Value of RGB features, Q D ,K D ,V D are the query matrix, key matrix and value matrix of deep features respectively, and MSA(·) is the multi-head self-attention mechanism;

[0137] The first mixed feature is obtained by analyzing the cross-input modal information through the RGB feature attention head The second mixed feature is obtained by analyzing the cross-input modal information through the deep feature attention head

[0138] Add the information output by the Transformer mixing block after the initial mixing to obtain the initial level image features The initial level image features The weights are calculated through the channel attention mechanism (i.e., channel attention block) and the spatial attention mechanism (i.e., spatial attention block) to obtain the hierarchical image features {F i |i=1,2,3,4}, specifically:

[0139]

[0140] Among them, F i is the hierarchical image feature, CA(·) is the channel attention mechanism, SA(·) is the spatial attention mechanism, ⊕ represents element-by-element addition, represents channel-by-channel multiplication, ⊙ represents element-by-element multiplication. F1 is the low-level image feature (i.e., the first-level depth feature With the first level RGB feature The first level image features obtained by fusion processing ), F4 is the high-level image feature (i.e. the fourth-level depth feature With the fourth level RGB feature The fourth level image features obtained by fusion processing ).

[0141] It should be understood that the number of features in the deep feature set is the same as the number of features in the multi-scale feature set, so as to fuse the features of two different modalities in a hierarchical order.

[0142] In the embodiment of the present invention, features of two different modalities are cross-fused to obtain fused features corresponding to different levels of RGB information and depth information.

[0143] Preferably, if Fig.10 As shown, the plurality of said hierarchical image features include high-level image features and low-level image features;

[0144] The step of extracting edge features from the plurality of hierarchical image features by the edge constraint module comprises:

[0145] The edge constraint module includes a plurality of upsampling blocks;

[0146] The high-level image features are upsampled by the first upsampling block to obtain edge high-level image features, the low-level image features are upsampled by the second upsampling block to obtain edge low-level image features, the edge high-level image features and the edge low-level image features are feature-connected to obtain initial edge features, the initial edge features are upsampled by the third upsampling block to obtain sampled edge features, and the sampled edge features are upsampled by the last upsampling block to obtain edge features.

[0147] Specifically, the edge constraint module includes 4 upsampling blocks, and the upsampling blocks include ordinary convolution layers and upsampling layers, specifically: the first upsampling block includes a second 1×1 convolution layer and a first upsampling layer, the second upsampling block includes a third 1×1 convolution layer and a second upsampling layer, the third upsampling block includes a third 3×3 convolution layer and a third upsampling layer, and the fourth upsampling block includes a fourth 1×1 convolution layer and a fourth upsampling layer.

[0148] The high-level image features are convolved by the first common convolution layer, and the high-level image features after convolution are upsampled by the first upsampling layer to obtain edge high-level image features. The low-level image features are convolved by the second common convolution layer, and the low-level image features after convolution are upsampled by the second upsampling layer to obtain edge low-level image features. The edge high-level image features and the edge low-level image features are feature-connected to obtain initial edge features, which are expressed as:

[0149]

[0150] in, is the initial edge feature, Cat is the feature connection operation, CU1 is the 1×1 convolution layer and upsampling operation, and F1 is the low-level image feature (i.e. the first-level image feature ), F4 is the high-level image feature (i.e. the fourth-level image feature ).

[0151] The initial edge feature is convolved by the third convolution layer, the initial edge feature after convolution is upsampled by the third upsampling layer to obtain a sampled edge feature, the sampled edge feature is convolved by the fourth convolution layer, and the sampled edge feature after convolution is upsampled by the fourth upsampling layer to obtain an edge feature, which is expressed as:

[0152]

[0153] Among them, F edge For edge features, CU3 is a 3×3 convolution layer with upsampling operations.

[0154] In the embodiment of the present invention, shallow features are fused with deep features, and the edges of salient targets are used as supervisory information in turn, so that the output edge information provides edge guidance for the model in the absence of an edge truth map.

[0155] Preferably, if Fig.11 As shown, the progressive decoder is used to decode the plurality of hierarchical image features and the edge features to obtain a significant target image, including:

[0156] The progressive decoder comprises a plurality of second common convolutional blocks and a plurality of common convolutional layers;

[0157] Convolution operations are performed on the hierarchical image features one by one through multiple second ordinary convolution blocks respectively, multiple ordinary convolutional hierarchical image features are added element by element to obtain initial salient features, the edge features are activated through an activation function to obtain activated edge features, the activated edge features are element-wise multiplied with the initial salient features to obtain initial edge salient features, the initial edge salient features are element-wise added with the initial salient features to obtain edge salient features, and ordinary convolution operations are performed on the edge salient features in sequence through multiple ordinary convolution layers to obtain a salient target image.

[0158] Specifically, the plurality of said hierarchical image features {F i |i=1,2,3,4} includes first-level image features (i.e., low-level image features), second-level image features, third-level image features, and fourth-level image features (i.e., high-level image features);

[0159] The progressive decoder includes a 3×3 convolution layer (i.e., the seventh 3×3 normal convolution layer), a 1×1 convolution layer (i.e., the fifth 1×1 normal convolution layer), and a plurality of second normal convolution blocks;

[0160] The fourth-level image feature is convolved through a second common convolution block, and the third-level image feature is convolved through a second common convolution block. The fourth-level image feature after convolution and the third-level image feature after convolution are added element by element to obtain the first initial intermediate feature. The first initial intermediate feature is upsampled to obtain the first intermediate feature, which is expressed as:

[0161] F 3_4 =TB(F3)⊕TB(F4),

[0162] Among them, F 3_4 is the first intermediate feature, F3 is the third-level image feature, F4 is the fourth-level image feature, TB(·) is the second ordinary convolution block (consisting of three 3×3 convolutional layers), and ⊕ is element-by-element addition;

[0163] The first intermediate feature is convolved through a second common convolution block, the second-level image feature is convolved through a second common convolution block, the first intermediate feature after convolution and the second-level image feature after convolution are added element by element to obtain the second initial intermediate feature, and the second initial intermediate feature is upsampled to obtain the second intermediate feature; the second intermediate feature is convolved through a second common convolution block, the first-level image feature is convolved through a second common convolution block, the second intermediate feature after convolution and the first-level image feature after convolution are added element by element to obtain the third initial intermediate feature, and the third initial intermediate feature is upsampled to obtain the third intermediate feature; the third intermediate feature is convolved through a second common convolution block, and the third intermediate feature after convolution is upsampled to obtain the initial significant feature, which is expressed as:

[0164] S f =δ ↑ (TB(δ ↑ (TB(F i )⊕TB(F i-1 )))),

[0165] Among them, S f is the initial salient feature, TB is the second ordinary convolution block, and F i is the i-th level image feature, δ ↑ is the upsampling operation, ⊕ is the element-by-element addition;

[0166] The edge features output by the edge constraint module are input into the progressive decoder to guide the highlighting of edge information in the initial salient features, and the salient features are obtained through a 3×3 convolution layer, which is expressed as:

[0167]

[0168] Among them, S t As a notable feature, S f is the initial salient feature, F edge is the edge feature, Conv3(·) is the convolution operation of 3×3 convolution kernel, Sig(·) is the Sigmoid activation function, ⊕ is element-by-element addition, is element-wise multiplication;

[0169] Through a 1×1 convolutional layer used as a projection head for salient features, the salient target image S is output o , expressed as:

[0170] S o =Conv1(S t ),

[0171] Among them, So is the salient target image, S t Conv1(·) is a convolution operation with a 1×1 convolution kernel.

[0172] The second common convolution block includes three 3×3 common convolution layers connected in series, that is, the fourth 3×3 common convolution layer, the fifth 3×3 common convolution layer and the sixth 3×3 common convolution layer are sequentially connected to form a second common convolution block; the convolution operation on the first intermediate feature through a second common convolution block and the convolution operation on the second-level image feature through a second common convolution block include:

[0173] The first intermediate feature is subjected to ordinary convolution operation through the fourth 3×3 ordinary convolution layer to obtain the initial convolution intermediate feature, the initial convolution intermediate feature is subjected to ordinary convolution operation through the fifth 3×3 ordinary convolution layer to obtain the convolution intermediate feature, and the convolution intermediate feature is subjected to ordinary convolution operation through the sixth 3×3 ordinary convolution layer to obtain the new convolution intermediate feature (i.e., the fourth-level image feature after convolution).

[0174] The process of performing convolution operations on the remaining second common convolution blocks is the same as the process of performing convolution operations on the second-level image features through one second common convolution block.

[0175] In the embodiment of the present invention, a progressive ladder decoder is used to progressively decode each layer of hierarchical image features, and features of different levels are utilized at different stages to improve the performance of the model.

[0176] Preferably, if Fig.10 As shown, the edge constraint module is used to extract the edge of the salient target image to obtain edge supervision features, including:

[0177] The edge of the salient target image obtained in the current training round is extracted through the sobel function to obtain the edge supervision feature, which is expressed as:

[0178]

[0179] in, is the edge supervision feature, S o is a salient target image, sobel(·) is a sobel function used to extract image edges;

[0180] In each round of training, the generation of edge supervision features is guided by the salient target image optimized in each round, that is, edge information is extracted from the salient target image. In this way, the model's ability to predict image edges is improved, resulting in a salient target map with a more complete filamentous structure and richer edge details.

[0181] It should be understood that the Sobel function (i.e., the Sobel operator) is a weighted difference of the grayscale values ​​of the four areas of up, down, left, and right of each pixel in the image, which reaches an extreme value at the edge to detect the edge, and is mainly used for edge detection.

[0182] In the embodiment of the present invention, the edge supervision feature is used to guide the edge constraint module to learn the edge information during training, and the cross entropy loss and the Dice similarity coefficient loss are subsequently used to constrain the training of the edge constraint module.

[0183] Preferably, the loss calculation is performed on the edge feature, the edge supervision feature, the salient target image and the pseudo label to obtain a total loss (i.e., a mixed loss), including:

[0184] The edge feature and the edge supervision feature are calculated by using an edge cross entropy loss function to obtain an edge cross entropy loss, and the edge cross entropy loss function is:

[0185]

[0186] in, is the marginal cross entropy loss, F edge is the edge feature, It is the edge supervision feature;

[0187] The pixel values ​​of the edge feature and the edge supervision feature are calculated by a similarity coefficient loss function to obtain a similarity coefficient loss, and the similarity coefficient loss function is:

[0188]

[0189] in, is the similarity coefficient loss, p i is the edge feature F edge The i-th pixel value, g i is the edge supervision feature The i-th pixel value of N1 is the total number of pixels of the salient target image;

[0190] The edge cross entropy loss and the similarity coefficient loss are summed to obtain the edge loss, and the edge loss summation expression is:

[0191]

[0192] in, is the edge loss, ω is the weighted binary coefficient;

[0193] The target cross entropy loss function is used to calculate the salient target image and the pseudo label to obtain the target cross entropy loss. The target cross entropy loss function is:

[0194]

[0195] in, is the target cross entropy loss, S o is a salient target image, and P is a pseudo label;

[0196] The edge pixel values ​​of the salient target image and the edge pixel values ​​of the pseudo label are calculated by an edge-related loss function to obtain an edge-related loss. The edge-related loss function is:

[0197]

[0198] in, is the edge-related loss, K is the salient target image S o The edge similarity relationship with the pseudo label P, W is the width of the pseudo label, and H is the height of the pseudo label;

[0199] The pixel values ​​of the salient target image and the pixel values ​​of the pseudo label are calculated by a regional correlation loss function to obtain a regional correlation loss, where the regional correlation loss function is:

[0200]

[0201] in, is the region-related loss, N2 is the total number of pixels of the pseudo-label;

[0202] The edge-related loss and the region-related loss are summed to obtain a hybrid enhancement loss, and the summation expression of the hybrid enhancement loss is:

[0203]

[0204] in, Enhance loss for mixing;

[0205] The target cross entropy loss and the hybrid enhancement loss are summed to obtain the significance loss, and the summation expression of the significance loss is:

[0206]

[0207] in, For significant losses;

[0208] The pixel value of the salient target image and the pixel value of the pseudo label are calculated by a pixel-level contrast loss function to obtain a pixel-level contrast loss. The pixel-level contrast loss function is:

[0209]

[0210] in, is the pixel-level contrast loss, M j,k is the salient target image S o The same area as the pixel of the pseudo label P, S j,k is the salient target image S o The pixel similarity value with the pseudo label P, j is the jth pixel of the pseudo label P, and k is the salient target image S o The kth pixel of

[0211] The pixel-level contrast loss, the saliency loss, and the edge loss are summed to obtain a total loss, and the summation expression of the total loss is:

[0212]

[0213] in, is the total loss, γ is the first learning parameter, set to 0.9, and β is the second learning parameter, set to 0.1.

[0214] In the embodiment of the present invention, the output edge features can effectively deal with the problem of missing edge information due to sparse labels in the weak supervision model. The pseudo labels are used to guide the salient target images output by the model to improve the model detection accuracy.

[0215] like Fig.12 As shown, an embodiment of the present invention provides an image saliency detection system based on multi-source weak supervision, including:

[0216] Import unit, used to import RGB images, depth images, text labels and point labels;

[0217] A labeling unit, used for matching the RGB image with the text label to obtain an initial pseudo label, and optimizing the initial pseudo label based on the point label to obtain a pseudo label;

[0218] A construction unit, used to construct a salient object detection model, wherein the salient object detection model includes a dual branch of a CNN encoder and a Swin-Transformer V2 encoder, and a hybrid attention module, an edge constraint module, and a progressive decoder respectively connected to the dual branches in sequence;

[0219] A prediction unit, configured to extract deep features from the depth image through the CNN encoder to obtain a deep feature set, extract channel features from the RGB image through the Swin-Transformer V2 encoder to obtain a multi-scale feature set, fuse the deep feature set and the multi-scale feature set through the hybrid attention module to obtain a plurality of different hierarchical image features, extract edge features from the plurality of hierarchical image features through the edge constraint module, and respectively decode the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a salient target image;

[0220] an optimization unit, configured to extract edges of the salient target image through the edge constraint module to obtain edge supervision features, perform loss calculation on the edge features, the edge supervision features, the salient target image and the pseudo-label to obtain a total loss, and optimize the salient target detection model through the total loss to obtain an optimized salient target detection model;

[0221] The application unit is used to detect the RGB image to be detected and the depth image to be detected by using the optimized salient target detection model to obtain a target salient target image.

[0222] The above-mentioned image saliency detection system based on multi-source weak supervision can refer to the implementation content and beneficial effects of the above-mentioned image saliency detection method based on multi-source weak supervision, which will not be repeated here.

[0223] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0224] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for image saliency detection based on multi-source weak supervision, characterized in that: The steps include: Import RGB images, depth images, text labels, and point labels; Matching the RGB image with the text label to obtain an initial pseudo label, and optimizing the initial pseudo label based on the point label to obtain a pseudo label; Constructing a salient object detection model, wherein the salient object detection model includes a dual branch of a CNN encoder and a Swin-Transformer V2 encoder, and a hybrid attention module, an edge constraint module, and a progressive decoder respectively connected to the dual branches in sequence; Performing deep feature extraction on the depth image through the CNN encoder to obtain a deep feature set, performing channel feature extraction on the RGB image through the Swin-Transformer V2 encoder to obtain a multi-scale feature set, performing fusion processing on the deep feature set and the multi-scale feature set through the hybrid attention module to obtain a plurality of different hierarchical image features, extracting edge features from the plurality of hierarchical image features through the edge constraint module, and respectively decoding the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a salient target image; Performing edge extraction on the salient target image by using the edge constraint module to obtain edge supervision features, performing loss calculation on the edge features, the edge supervision features, the salient target image and the pseudo-label to obtain a total loss, and optimizing the salient target detection model by using the total loss to obtain an optimized salient target detection model; The optimized salient target detection model is used to detect the RGB image to be detected and the depth image to be detected to obtain a target salient target image.

2. The image saliency detection method according to claim 1, characterized in that: The step of matching the RGB image with the text label to obtain an initial pseudo label includes: Perform feature extraction on the RGB image and the text label through the CLIP model to obtain encoding attention weights, image features, and text features; The probability distribution of the image features and the text features is calculated by the softmax function to obtain a foreground class score and a background class score. The foreground class score and the background class score are calculated by a weight calculation expression to obtain multiple pixel point category weights. The weight calculation expression is: in, is the category weight, Z is the total pixel value, y c is the pixel fraction of foreground class c, is the pixel coordinate on the kth channel, s c is the foreground class c score, y c′ is the pixel fraction of background class c′, s c′ is the background class c′ score; The weighted summation of the multiple pixel category weights is performed to obtain an initial category activation map, the mask extraction is performed on the initial category activation map to obtain a category mask, and the encoding attention weight, the initial category activation map and the category mask are calculated by a weight fusion expression to obtain a category activation map, and the weight fusion expression is: Among them, M t is the category activation map, B c is the class mask, W at To encode the attention weight, M c is the initial category activation map, vec(·) is the matrix vectorization, and ⊙ is the element-by-element multiplication; The pixels of the class activation map are filtered according to the set filtering parameters to obtain initial pseudo labels.

3. The image saliency detection method according to claim 1, characterized in that: The point labels include a foreground target point image and a background target point image; The step of optimizing the initial pseudo-label based on the point label to obtain the pseudo-label includes: Extract features from the RGB image using a Sobel algorithm to obtain a rough edge image, and perform pixel filling processing on the rough edge image according to the foreground target point image using a Flood Fill algorithm to obtain a foreground target activation image; The foreground target activation image and the initial pseudo-label are added to obtain a background target image, and the background target image is pixel-filled according to the background target point image using the FloodFill algorithm to obtain a background target activation image, and the background target activation image is refined using a conditional random field algorithm to obtain a pseudo-label.

4. The image saliency detection method according to claim 1, characterized in that: The extracting depth features of the depth image by the CNN encoder to obtain a depth feature set includes: The CNN encoder includes a point-by-point convolution layer, a standard normalization layer, an activation function layer, a first common convolution block and a plurality of depth-separable convolution blocks connected in series, wherein the first common convolution block is connected to the first depth-separable convolution block, the remaining depth-separable convolution blocks are connected in sequence, and the last depth-separable convolution block is connected to the point-by-point convolution layer, the standard normalization layer and the activation function layer in sequence; Performing a convolution operation on the depth image through the first common convolution block to obtain common convolution features; The first depth-separable convolution block performs a convolution operation on the common convolution feature to obtain a first-level depth feature, and the second depth-separable convolution block performs a convolution operation on the first depth feature to obtain a second-level depth feature, and so on, until the last depth-separable convolution block performs a convolution operation on the output of the previous depth-separable convolution block to obtain a new-level depth feature; Performing point-by-point convolution operation on the new-level depth feature through the point-by-point convolution layer, normalizing the new-level depth feature after point-by-point convolution through the standard normalization layer, and activating the normalized new-level depth feature through the activation function layer to obtain the optimal level depth feature; The deep feature set is composed of the output of each depth-separable convolutional block and the optimal level deep features.

5. The image saliency detection method according to claim 1, characterized in that: The step of extracting channel features from the RGB image by using the Swin-Transformer V2 encoder to obtain a multi-scale feature set includes: The Swin-Transformer V2 encoder includes a plurality of serially connected Transformer basic blocks connected to a partitioning layer; The RGB image is divided into blocks through the division layer to obtain a plurality of image blocks, and the plurality of image blocks are linearly mapped to obtain window embedding features; The first Transformer basic block performs a window multi-head self-attention operation on the window embedding feature to obtain a first initial window attention weight, adds the first initial window attention weight to the window embedding feature to obtain a first window attention weight, performs a cross-window multi-head self-attention operation on the first window attention weight to obtain a first initial cross-window attention weight, adds the first initial cross-window attention weight to the first window attention weight to obtain a first cross-window attention weight, performs feature extraction on the first cross-window attention weight through a multi-layer perceptron to obtain a first perceptual feature, adds the first perceptual feature to the first cross-window attention weight to obtain a first-level RGB feature, and performs a multi-layer perceptron on the first cross-window attention weight to obtain a first-level RGB feature. Perform a window multi-head self-attention operation on the cross-window attention weight to obtain a second initial window attention weight, add the second initial window attention weight to the first cross-window attention weight to obtain a second window attention weight, perform a cross-window multi-head self-attention operation on the second window attention weight to obtain a second initial cross-window attention weight, add the second initial cross-window attention weight to the second window attention weight to obtain a second cross-window attention weight, perform feature extraction on the second cross-window attention weight through a multi-layer perceptron to obtain a second perceptual feature, add the second perceptual feature to the second cross-window attention weight to obtain a second-level RGB feature, and so on, until the last Transformer basic block obtains the last-level RGB feature; The hierarchical RGB features output by each Transformer basic block constitute a multi-scale feature set.

6. The image saliency detection method according to claim 1, characterized in that: The depth feature set includes multiple levels of depth features, the multi-scale feature set includes multiple levels of RGB features, and the multiple levels of depth features correspond to the multiple levels of RGB features in one-to-one correspondence; The hybrid attention module is used to fuse the deep feature set and the multi-scale feature set to obtain multiple different levels of image features, including: The hybrid attention module includes a Transformer hybrid block, a channel attention block, and a spatial attention block connected in sequence; The Transformer mixing block is used to cross-calculate any level of depth feature and the corresponding level of RGB feature to obtain a first mixed feature and a second mixed feature, the first mixed feature and the second mixed feature are added to obtain an initial level image feature, the channel information of the initial level image feature is weighted by the channel attention block to obtain a channel attention weight, the channel attention weight is multiplied element-by-element by the initial level image feature to obtain a channel-level image feature, the spatial information of the channel-level image feature is weighted by the spatial attention block to obtain a spatial attention weight, the spatial attention weight is multiplied channel-by-channel by the initial level image feature to obtain a spatial level image feature, the spatial level image feature is added element-by-element by the initial level image feature to obtain a level image feature, and in this process, all levels of depth features and all levels of RGB features are processed to obtain multiple different levels of image features.

7. The image saliency detection method according to claim 1, characterized in that: The plurality of hierarchical image features include high-level image features and low-level image features; The step of extracting edge features from the plurality of hierarchical image features by the edge constraint module comprises: The edge constraint module includes a plurality of upsampling blocks; The high-level image features are upsampled by the first upsampling block to obtain edge high-level image features, the low-level image features are upsampled by the second upsampling block to obtain edge low-level image features, the edge high-level image features and the edge low-level image features are feature-connected to obtain initial edge features, the initial edge features are upsampled by the third upsampling block to obtain sampled edge features, and the sampled edge features are upsampled by the last upsampling block to obtain edge features.

8. The image saliency detection method according to claim 1, characterized in that: The step of respectively decoding the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a significant target image includes: The progressive decoder comprises a plurality of second common convolutional blocks and a plurality of common convolutional layers; Convolution operations are performed on the hierarchical image features one by one through multiple second ordinary convolution blocks respectively, multiple ordinary convolutional hierarchical image features are added element by element to obtain initial salient features, the edge features are activated through an activation function to obtain activated edge features, the activated edge features are element-wise multiplied with the initial salient features to obtain initial edge salient features, the initial edge salient features are element-wise added with the initial salient features to obtain edge salient features, and ordinary convolution operations are performed on the edge salient features in sequence through multiple ordinary convolution layers to obtain a salient target image.

9. The image saliency detection method according to claim 1, characterized in that: The loss calculation of the edge feature, the edge supervision feature, the salient target image and the pseudo label to obtain a total loss includes: The edge feature and the edge supervision feature are calculated by using an edge cross entropy loss function to obtain an edge cross entropy loss, and the edge cross entropy loss function is: in, is the marginal cross entropy loss, F edge is the edge feature, It is the edge supervision feature; The pixel values ​​of the edge feature and the edge supervision feature are calculated by a similarity coefficient loss function to obtain a similarity coefficient loss, and the similarity coefficient loss function is: in, is the similarity coefficient loss, p i is the i-th pixel value of the edge feature, g i is the i-th pixel value of the edge supervision feature, and N1 is the total number of pixels of the salient target image; The edge cross entropy loss and the similarity coefficient loss are summed to obtain the edge loss; The target cross entropy loss function is used to calculate the salient target image and the pseudo label to obtain the target cross entropy loss. The target cross entropy loss function is: in, is the target cross entropy loss, S o is a salient target image, and P is a pseudo label; The edge pixel values ​​of the salient target image and the edge pixel values ​​of the pseudo label are calculated by an edge-related loss function to obtain an edge-related loss. The edge-related loss function is: in, is the edge-related loss, K is the edge similarity relationship, W is the width of the pseudo-label, and H is the height of the pseudo-label; The pixel values ​​of the salient target image and the pixel values ​​of the pseudo label are calculated by a regional correlation loss function to obtain a regional correlation loss, where the regional correlation loss function is: in, is the region-related loss, N2 is the total number of pixels of the pseudo-label; The edge-related loss and the region-related loss are summed to obtain a hybrid enhancement loss, and the target cross entropy loss and the hybrid enhancement loss are summed to obtain a significance loss; The pixel value of the salient target image and the pixel value of the pseudo label are calculated by a pixel-level contrast loss function to obtain a pixel-level contrast loss. The pixel-level contrast loss function is: in, is the pixel-level contrast loss, M j,k For the same pixel area, S j,k is the pixel similarity value, j is the jth pixel of the pseudo label, and k is the kth pixel of the salient target image; The pixel-level contrast loss, the saliency loss, and the edge loss are summed to obtain a total loss.

10. An image saliency detection system based on multi-source weak supervision, characterized in that: include: Import unit, used to import RGB images, depth images, text labels and point labels; A labeling unit, used for matching the RGB image with the text label to obtain an initial pseudo label, and optimizing the initial pseudo label based on the point label to obtain a pseudo label; A construction unit, used to construct a salient object detection model, wherein the salient object detection model includes a dual branch of a CNN encoder and a Swin-Transformer V2 encoder, and a hybrid attention module, an edge constraint module, and a progressive decoder respectively connected to the dual branches in sequence; A prediction unit, configured to extract deep features from the depth image through the CNN encoder to obtain a deep feature set, extract channel features from the RGB image through the Swin-Transformer V2 encoder to obtain a multi-scale feature set, fuse the deep feature set and the multi-scale feature set through the hybrid attention module to obtain a plurality of different hierarchical image features, extract edge features from the plurality of hierarchical image features through the edge constraint module, and respectively decode the plurality of hierarchical image features and the edge features through the progressive decoder to obtain a salient target image; an optimization unit, configured to extract edges of the salient target image through the edge constraint module to obtain edge supervision features, perform loss calculation on the edge features, the edge supervision features, the salient target image and the pseudo-label to obtain a total loss, and optimize the salient target detection model through the total loss to obtain an optimized salient target detection model; The application unit is used to detect the RGB image to be detected and the depth image to be detected by using the optimized salient target detection model to obtain a target salient target image.

Images