A Graffiti-Based Weakly Supervised Saliency Target Detection Method

Abstract

This case relates to a graffiti-based weakly supervised salient object detection method based on complementary graph inference networks, aiming to address the limitation of existing weakly supervised salient object detection methods in recovering complete salient object structures from sparse graffiti annotations. The method inputs the image to be segmented into a trained object detection model and outputs the object segmentation result. The object detection model is weakly supervised and trained using graffiti annotations. The steps include: extracting multi-scale features from the sample image to obtain basic edge features, basic salient features, and coarse-grained salient features; obtaining corresponding graph representations based on the basic edge features and basic salient features; after cross-complementing the representations, using a graph convolutional network to propagate the representation information to obtain enhanced edge features and enhanced salient features; refining the enhanced edge features to obtain salient edge features; and fusing the salient edge features, enhanced salient features, and coarse-grained salient features to obtain the object segmentation result.

Description

Technical Field

[0001] This case involves the field of weakly supervised salient object detection, and in particular, it relates to a graffiti-style weakly supervised salient object detection method based on complementary graph reasoning networks. Background Technology

[0002] In weakly supervised salient object detection, when using convolutional neural networks, contextual information can be transformed into more expressive features. For example, Piao et al. proposed a saliency network with multiple filters for image-level labeling, extracting and filtering more accurate saliency cues from noisy pseudo-labels. Yu et al. proposed a single-round end-to-end training method for weakly supervised salient object detection using doodle annotations, introducing gated conditional random field loss to propagate labels to unlabeled regions based on image features and pixel distance, thereby predicting overall salient regions with consistent object structures. However, this method suffers from incomplete predicted salient regions because the doodle annotations are very sparse. Secondly, low-contrast regions near the boundaries of salient objects exhibit strong uncertainty, making it difficult to correctly classify hard pixels, ultimately resulting in saliency maps with blurred boundaries. Some methods generate pseudo-labels to obtain reliable annotation information, but this merely shifts the problem of sparse annotations. Pseudo-labels often contain a lot of noise, and the overall scheme is complex and difficult to reproduce. Furthermore, some works have introduced edge labels for supervised training. However, such methods only utilize edge information to guide salient object segmentation without considering the relationship between the two. One potential issue is that this increases the model's dependence on edge detection accuracy, leading to the loss of more global information. Moreover, the introduced edge features are likely to contain noise, which could even degrade the performance of the salient object detection model, causing it to get stuck in a local optimum. Summary of the Invention

[0003] To address the problem that existing weakly supervised salient object detection methods cannot recover the complete salient object structure from doodle annotations, this paper proposes a doodle-based weakly supervised salient object detection method based on complementary graph reasoning networks. By capturing the inherent correspondence between pixels and utilizing its relationship with global information, accurate object detection is achieved. By incorporating a dense fusion strategy to integrate contextual information at different levels, such as corrected edge features, enhanced salient features, and coarse-grained salient features, the robustness of object detection is improved. Furthermore, by allocating the number of graph representation centers on demand, the scheme becomes more universal.

[0004] To achieve the above technical objectives, the specific technical solution of this case is as follows.

[0005] Firstly, this paper proposes a graffiti-based weakly supervised salient object detection method based on complementary graph reasoning networks. This method effectively utilizes contextual information and considers its relationship with global information during object detection, overcoming the shortcomings of existing weakly supervised salient object detection methods that cannot recover the complete salient object structure from sparse graffiti annotations and lack consideration for the relationship between local and global features. Specifically, the method inputs the image to be segmented into a trained object detection model and outputs the object segmentation result; wherein: the object detection model is weakly supervised by graffiti annotations, and the steps include:

[0006] Multi-scale features are extracted from sample images to obtain basic edge features, basic saliency features, and coarse-grained saliency features;

[0007] Based on basic edge features and basic saliency features, corresponding graph representations are obtained. After the representations are cross-complemented, the representation information is propagated using a graph convolutional network to obtain enhanced edge features and enhanced saliency features.

[0008] The enhanced edge features are then modified to obtain salient edge features;

[0009] The salient edge features, enhanced salient features, and coarse-grained salient features are fused together to obtain the target segmentation result;

[0010] The training loss of the object detection model is composed of the gated conditional random field loss L. lsc Edge prediction loss L e , saliency prediction loss for graffiti-annotated pixel regions L pce constitute.

[0011] In the above technical solution, the acquisition steps for basic edge features, basic saliency features, and coarse-grained saliency features include, in one embodiment, the steps as follows:

[0012] A five-layer convolutional neural network is used to extract image features, and the features f of each layer are obtained sequentially. i (i = 1, 2, 3, 4, 5);

[0013] The basic edge features are obtained by resampling the low-level features f1 and f2 and then concatenating them. The basic saliency features are obtained by resampling the layer features f3, f4 and f5.

[0014] Coarse-grained significance features are obtained based on the high-level feature f5.

[0015] In the above technical solutions, enhancing edge features and enhancing saliency features, one implementation includes the following steps:

[0016] Based on fundamental edge features and fundamental saliency features, edge graph representation Z is obtained using graph projection. E Significance plot characterizes Z S ;

[0017] Characterize the edge map Z E Significance plot characterizes Z S Spatial location encoding is performed separately, and the edge map representation Z′ is obtained by cross-complementing the representations of the two using a multilayer perceptron model. E Significance plot characterizes Z′ S The representation information is then propagated through a graph convolutional network to obtain an enhanced edge graph representation. Enhanced saliency graph representation Then, back projection is used to restore the original spatial coordinates of the two objects, which are then used to enhance edge features. and enhanced saliency features

[0018] In the above technical solutions, one implementation rule for representing the intersection is:

[0019]

[0020] Where: Relationship matrix G is obtained by calculating the correspondence between graph representations. e (·) and G S (·) represents a combination of 1D convolution, batch normalization, and the ReLU activation function. GCU1 is the first representation crossover unit, and GCU2 is the second representation crossover unit. Indicates that Z E After initial processing by the MLP layer, edge graph representation is performed before the intersection. Indicates that Z S After initial processing by the MLP layer, a saliency graph representation is performed before representation crossover. D is the channel dimension, V is the preset number of cluster centers in the graph projection, and R is the edge graph representation Z′. E Or a significance graph characterizing Z′ S The number field.

[0021] In one implementation of the above technical solution for propagating representation information using graph convolutional networks, the computational method employed is as follows:

[0022]

[0023] In the formula: GCN(·) is a graph convolutional network, σ(·) is a nonlinear activation function, and the similarity matrix A E =Z′ E T ×Z′ E A S =Z′S T ×Z′ S , and It is a set of learnable parameters.

[0024] In the above technical solutions, edge features are enhanced. and enhanced saliency features The acquisition steps include, in one embodiment, the acquisition steps include:

[0025] Based on basic edge features f e Obtain the corresponding soft allocation matrix Q E Based on the fundamental significance feature f S Obtain the corresponding soft allocation matrix Q S ;

[0026] The original spatial coordinate features are obtained by performing back projection calculations using the following formula:

[0027]

[0028] Where: R H×W×D It is the feature dimension space of the original spatial coordinates, where H is the height of the edge feature or salient feature, W is the width of the edge feature or salient feature, and D represents the channel dimension of the edge feature or salient feature.

[0029] In the above technical solution, the acquisition steps for salient edge features, in one embodiment, include:

[0030] Will enhance edge features The feature F′ is obtained after convolution processing. e And calculate feature F′ e The channel weight d is then used to obtain the corrected edge features.

[0031]

[0032]

[0033] In the formula: C3 and C4 represent convolutional layers, GAP(·) represents global average pooling, ReLU(·) is a non-linear activation function, and δ(·) represents the sigmoid function. Indicates pixel-by-pixel multiplication;

[0034] Use corrected edge features Obtain enhanced edge features It is then connected to the edge features of the input image extracted and corrected using the Canny operator, and then the number of channels is adjusted using convolutional layers to generate salient edge features f. E ,Right now:

[0035]

[0036] In the formula: Canny(·) is the Canny edge extraction operator, I represents the input image, and Concat(·) is the channel-wise concatenation operation. This represents adding pixels one by one.

[0037] In the above technical solution, one implementation method of integration includes the following steps:

[0038] Based on enhanced saliency features Channel adjustments were made to obtain the enhanced significance feature f. S ;

[0039] This will enhance the saliency feature f S Significant marginal features f E and coarse-grained significance feature S C The calculation rules for the dense fusion strategy are as follows:

[0040]

[0041]

[0042] S r =Conv(Concat(Conv(Up(S1)),S2))

[0043] In the formula: Conv(·) represents a convolutional layer, and Up(·) represents an upsampling operation. This indicates pixel-wise multiplication, and Concat(·) indicates channel concatenation operation.

[0044] In the above technical solution, one method for calculating the training loss of the object detection model is as follows:

[0045] L total =L pce +L e +αL lsc

[0046]

[0047]

[0048]

[0049] In the formula:

[0050] L lsc Let L be the loss function of the gated conditional random field. pce For partial cross-entropy loss, L eFor edge loss;

[0051] D(i,j)=|S i -S j |,S i To predict the significance score of pixel i in the image, S j To predict the significance score of pixel j in the image, D(i,j) is the difference in significance scores between pixel i and pixel j in the predicted image, and K... i is the window size for the energy diffusion of pixel i, and F(i,j) is the similarity energy function; where, the significance score is obtained by performing regression processing on the fusion result and the graffiti label, and then transforming the regression result using the softmax function;

[0052] J s denoted by doodle, E is the ground truth region of the edge, e is the edge prediction region, y is the saliency ground truth region, and x is the saliency prediction region.

[0053] Secondly, this case proposes a salient object detection system, which inputs the image to be segmented into a trained object detection model and outputs the object segmentation result;

[0054] The target detection model includes a basic network module, a graph inference network module, a correction module, and a fusion module;

[0055] The basic network module is configured to extract multi-scale features from sample images, thereby obtaining basic edge features, basic saliency features, and coarse-grained saliency features;

[0056] The graph reasoning network module is configured to obtain corresponding graph representations based on basic edge features and basic saliency features. After the representations are cross-complemented, the representation information is propagated using a graph convolutional network to obtain enhanced edge features and enhanced saliency features.

[0057] The correction module is configured to correct the enhanced edge features to obtain salient edge features;

[0058] The fusion module is configured to fuse salient edge features, enhanced salient features, and coarse-grained salient features to obtain the target segmentation result;

[0059] Wherein: the training loss of the object detection model is composed of the gated conditional random field loss L lsc Edge prediction loss L e , saliency prediction loss for graffiti-annotated pixel regions L pce constitute.

[0060] The beneficial technical effects of this case are as follows:

[0061] 1) Using salient object edge context information to assist weakly supervised salient object detection, more accurate weakly supervised salient object detection is achieved by leveraging the dependency between edges and salient object graph representations; this is achieved by using salient edge features f E Enhance saliency features f S and coarse-grained significance feature S C Dense fusion is performed to integrate multiple contextual information such as graph representation, edge features, and salient object features, which improves the robustness of the method and overcomes the uncertainty caused by the fact that the receptive field of pixels near the boundary often contains both salient objects and background.

[0062] 2) Extract the boundary features of salient objects in the shallow layer of the network to enhance the network’s attention to the boundary region and provide clues for subsequent salient object inference, so as to recover the overall structure of salient objects from sparse annotations.

[0063] 3) Using graph projection to achieve clustering based on the similarity of feature points enables the graph reasoning process to directly capture the global long-distance correspondence between pixels, which is beneficial for extending the local central graffiti point to the whole object.

[0064] 4) By setting the number of center nodes when projecting the graph, the scheme can be made more universal. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 A schematic diagram of an implementation framework for a graffiti-style weakly supervised salient object detection method based on complementary graph reasoning networks;

[0067] Figure 2 A schematic diagram illustrating the implementation process of information interaction using graph representation;

[0068] Figure 3 A schematic diagram of an implementation process for an edge-guided module;

[0069] Figure 4 A schematic diagram illustrating the implementation of a dense integration strategy;

[0070] Figure 5 ,exist Figure 1 A visual diagram comparing the method under the implementation framework with various weakly supervised saliency target detection methods. Detailed Implementation

[0071] Salience object detection methods aim to detect and segment the most salient objects in an image and are widely used in various visual tasks, such as object tracking, information retrieval, semantic segmentation, and image processing. Salience object detection methods include fully supervised and weakly supervised methods. This scheme primarily addresses the issue of incomplete salient region prediction and blurred boundaries in weakly supervised salient object detection methods using doodle annotations, where existing technologies lack effective solutions. This scheme inputs the image to be segmented into a trained object detection model and outputs the object segmentation result; wherein the object detection model is weakly supervised and trained using doodle annotations, the steps of which include:

[0072] Extract multi-scale features from sample images, and then obtain basic edge features f. e Basic significance features f s Coarse-grained saliency characteristics S C ;

[0073] Based on basic edge features f e and basic significance features f s Each element obtains its corresponding graph representation. After the representations are cross-complemented, the representation information is propagated using a graph convolutional network, thereby obtaining enhanced edge features. and enhanced saliency features

[0074] Will enhance edge features Make corrections to obtain salient edge features f E ;

[0075] Significant edge features f E Enhance saliency features f S and coarse-grained significance feature S C Perform fusion to obtain segmentation result S r ;

[0076] The training loss of the object detection model is composed of the gated conditional random field loss L. lsc Edge prediction loss L e , saliency prediction loss for graffiti-annotated pixel regions L pce constitute.

[0077] The following description, in conjunction with the accompanying drawings, clearly and completely describes how the technical solution of this case is implemented. Obviously, the described embodiments are only a part of the embodiments of this case, and not all of them. Based on the embodiments in this case, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0078] Figure 1A schematic diagram of the working framework of the object detection model is given. It takes the image to be segmented as input, and processes it through the basic network module, the graph inference network module, the correction module and the fusion module to output the object segmentation result.

[0079] (I) Basic Network Module

[0080] The basic network module is used to extract multi-scale features from the image to be segmented. For example, a five-layer convolutional neural network is used to extract image features, resulting in multi-scale features f. i (i = 1, 2, 3, 4, 5), f i The image features extracted by the i-th layer of the convolutional neural network. Since detail information is gradually diluted while semantic information is gradually enhanced during the top-down convolutional feature extraction process, the low-level features f1 and f2 are resampled and concatenated to obtain the basic edge feature f. e The basic saliency feature f is obtained by resampling the layer features f3, f4, and f5 and then concatenating them. s The high-level feature f5 is processed using spatial pyramid pooling to obtain the coarse-grained saliency feature S. C When reducing the accuracy of target segmentation, layer feature f5 can also be directly used as a coarse-grained saliency feature S. C The selection of the base network takes into account both efficiency and accuracy; for example, the base network adopts the VGGNet-16 network.

[0081] (II) Graph Reasoning Network Module

[0082] First, the basic edge features and basic saliency features obtained from the basic network modules are modeled as edge graph representations and saliency graph representations.

[0083] One implementation involves first using graph projection to cluster based on the similarity of feature points. Specifically, the basic features are first modified into a graph relation matrix G = (V, E), where vertices are denoted as V and edges as E. Then, a preset number of cluster center nodes is used, and the weight of each pixel feature to the preset cluster nodes is calculated, thus obtaining a soft assignment matrix Q. By setting the number of center nodes during graph projection, the scheme can be made universal. Finally, the soft assignment matrix and the feature map are weighted and summed to obtain the graph representation of the corresponding features. Using graph projection to cluster based on the similarity of feature points allows the graph reasoning process to directly capture the global long-distance correspondence between pixels, which is beneficial for extending local center points to the entire object.

[0084] The soft allocation matrix Q is calculated as follows:

[0085]

[0086] in, The pixel feature x at pixel position nn The distance to the k-th cluster center, ω i and σ i It is a learnable parameter, and the value of i ranges from 1 to k.

[0087] Cluster center is a hyperparameter, and the essence of the soft assignment matrix is ​​to calculate the relationship between feature points and cluster centers ω. i The Gaussian radial basis distance is calculated, and the normalized weights are derived.

[0088] The graphical representation calculation method corresponding to the feature is as follows:

[0089]

[0090]

[0091] In the formula: z′ k It is the eigenvector x n With vertex parameter ω k The weighted average of the residuals between them, z′ k Further L2 normalization yields the eigenvector z of node k. k , z k This constitutes the graph representation Z∈R D×V The k-th column, D represents the channel dimension, and V represents the preset number of cluster centers.

[0092] Using the above method, the basic edge features f are obtained respectively. e Basic significance features f s Edge graph representation of Z E Significance plot characterizes Z S .

[0093] Next, see Figure 2 Z represents the edge graph E Significance plot characterizes Z S Spatial location encoding is performed, and a multilayer perceptron (MLP) model is used to achieve cross-complementarity of graph representations between the two. The correspondence between the graph representations is calculated to obtain the relation matrix. The edge graph representation Z′ after semantic cross-transformation E Significance plot characterizes Z′ S .

[0094] Relationship matrix The calculation method is as follows:

[0095]

[0096] Among them, f bmm This represents batch matrix multiplication. Indicates that Z EAfter initial processing by the MLP layer, edge graph representation is performed before the intersection. Indicates that Z S After initial processing by the MLP layer, a saliency graph representation is performed before the characterization crossover.

[0097] Z E and Z S The crossover rule is:

[0098]

[0099] in:

[0100] G e (·) and G s (·) represents a combination of 1D convolution, batch normalization, and the ReLU activation function. GCU1 is the first representation cross-unit, GCU2 is the second representation cross-unit, D is the channel dimension, V is the preset number of cluster centers in the graph projection, and R is the edge graph representation Z′. E Or a significance graph characterizing Z′ S The term "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0101] Then, Z′ is further propagated using a graph convolutional network. E and Z′ S The information above is used to obtain an enhanced edge map representation. and enhanced saliency plot representation

[0102] The method for calculating the propagation of representation information in graph convolutional networks is as follows:

[0103]

[0104] In the formula: GCN(·) is a graph convolutional network, σ(·) is a nonlinear activation function, and the similarity matrix A E =Z′ E T ×Z′ E A S =Z′ S T ×Z′ S , and It is a set of learnable parameters.

[0105] The graph representation obtained after propagating representation information through a graph convolutional network. Using the soft allocation matrix Q as input, the two are multiplied by graph backprojection, and the graph representation is reweighted to obtain the feature map of the original spatial coordinates—enhanced edge features. and enhanced saliency features Specifically:

[0106] Based on basic edge features f e Obtain the corresponding soft allocation matrix Q E Based on the fundamental significance feature f s Obtain the corresponding soft allocation matrix Q S ;

[0107] The original spatial coordinate features are obtained by performing back projection calculations using the following formula:

[0108]

[0109] Where: R H×W×D It is the feature dimension space of the original spatial coordinates, where H is the height of the edge feature or salient feature, W is the width of the edge feature or salient feature, and D represents the channel dimension of the edge feature or salient feature.

[0110] Thus, the above process first converts the three-dimensional basic edge features and basic saliency features into corresponding two-dimensional graph representations. Then, at the two-dimensional level, it learns the feature cross-complementation and representation information propagation between the two. Finally, it projects the learned graph representation back into three-dimensional space to obtain the original spatial coordinates, thus obtaining the enhanced three-dimensional edge features. and enhanced saliency features

[0111] (III) Correction Module

[0112] The correction module will obtain enhanced edge features After correction, the salient edge feature f is obtained. E .

[0113] During the training phase, edge features will be enhanced. Edge feature regression is performed using the ground truth edge features. See also Figure 3 First, a reliable edge detection network, RCF, is used to extract the edge images of salient targets, which are then used as an additional pre-training set. Next, residual channel attention is used to assign different weights to different channels of the feature map to focus on edge interest points and improve the reliability of boundary information. Channel weights d, and edge feature correction are then implemented. The calculation method is as follows:

[0114] d=δ(C4(ReLU(C3(GAP(F′ e )))))

[0115]

[0116] Where C3 and C4 represent convolutional layers, GAP(·) represents global average pooling, ReLU(·) is a non-linear activation function, δ(·) represents the sigmoid function, and F′ e It enhances edge features Obtained through convolution. This indicates pixel-by-pixel multiplication.

[0117] Obtaining corrected edge features Then, use corrected edge features. Correcting and enhancing edge features It is then connected to the edge features extracted from the input image using the Canny operator, and then the number of channels is adjusted using convolutional layers to generate salient edge features f. E ,Right now:

[0118]

[0119] In the formula: Canny(·) is the Canny edge extraction operator, I represents the input image, and Concat(·) is the channel-wise concatenation operation. This represents adding pixels one by one.

[0120] (iv) Integration Module

[0121] The fusion module will enhance saliency features Significant marginal features f E It is fused with the coarse-grained saliency feature Sc previously obtained based on layer feature f5. In implementation, a dense fusion strategy was employed, such as... Figure 4 As shown.

[0122] Before fusion, enhance saliency features. Channel adjustments were made to obtain the enhanced significance feature f. S This is then densely fused with salient edge features and coarse-grained salient features. The calculation rules for the dense fusion strategy are as follows:

[0123]

[0124]

[0125] S′ c =Conv(Concat(Conv(Up(S1)),S2))

[0126] In the formula: Conv(·) represents a convolutional layer, This indicates pixel-wise multiplication, Concat(·) indicates channel concatenation, and f EFor salient edge features, f S To enhance the saliency feature, S C For coarse-grained significance features, Up(·) denotes an upsampling operation. For example, when using a 2x upsampling (see...), Figure 4 When performing a certain operation, it is possible to achieve a smaller computational load without losing key feature details.

[0127] The obtained fusion result S' C Further processing using residual channel attention modules and convolutional layers yields a visual target segmentation result S. r .

[0128] During training, the residual channel attention component fuses the results with the graffiti label regression.

[0129] In this study, the softmax function is used to convert the regression results into significance scores during training, while the partial cross-entropy function and the gated conditional random field loss function are used to calculate the loss for backpropagation.

[0130] Gated Conditional Random Field Loss Function L lsc Specifically, it includes:

[0131]

[0132] Where, D(i,j)=|S i -S j |,S i To predict the significance score of pixel i in the image, S j To predict the significance score of pixel j in the image, D(i,j) is the difference in significance scores between pixel i and pixel j in the predicted image, and K... i Let F(i,j) be the window size for energy diffusion of pixel i, and let F(i,j) be the similarity energy function, expressed as follows:

[0133]

[0134] P(·) represents the position of the pixel, I(·) represents the RGB value of the pixel, 1 / ω represents the normalization weight, and σ P The hyperparameter σ represents the Gaussian kernel scale of the pixel location. I The hyperparameter representing the Gaussian kernel scale of the pixel's RGB values, ||·|| 2 It is an L2 norm.

[0135] L pce and L e Let represent the partial cross-entropy loss and the edge loss, respectively. Their calculation process is as follows:

[0136]

[0137]

[0138] In the formula: J s y represents the pixel region marked by the graffiti, E is the edge ground truth region, e is the edge prediction region, y is the saliency ground truth region, x is the saliency prediction region, and i is the pixel label.

[0139] The final loss function L of the entire framework total For edge loss L e Partial cross-entropy loss L pce and gated random field loss L lsc The sum of the three parts:

[0140] L total =L pce +L e +αL lsc .

[0141] In the formula: α is an empirical value. For example, when α is 2, a well-performing target detection model can be obtained.

[0142] The above implementation scheme was tested on four well-known public datasets: ECSSD, DUTS-TE, DUT-OMRON, and HKU-IS. The ECSSD dataset consists of 1000 images with complex scenes from MSRA. DUTS-TE is currently the largest saliency evaluation test set, containing 5019 images, all collected from the ImageNet DET test set and the SUN dataset. The DUT-OMRON dataset includes 5168 high-quality images manually selected from over 140,000 images, each test image having one or more salient objects and a relatively complex background. HKU-IS contains 4447 pixel-level labeled images that meet at least one of the following conditions: at least one salient object touches the boundary of the image; multiple unconnected salient objects exist; and color contrast is less than 0.7. Consistent with most methods, this scheme uses MAE and F-measure as evaluation metrics. A smaller MAE value indicates better model performance, while the F-measure considers both recall and precision, denoted by F. β Refers to, F β A higher value indicates better model performance.

[0143] Table 1 shows the comparisons of this invention with other methods on the ECSSD validation set in MAE and F. β Comparison of indicators.

[0144] Table 1

[0145]

[0146] Table 1 shows that the proposed solution achieved good accuracy on this dataset, even with an MAE of 0.05, it still achieved a certain improvement, and the F... β The results are nearly identical to the optimal outcome. It's worth noting that the improvement of this approach is not only reflected in quantitative data comparisons, but also in the significant improvement in the comparison of visual results. Figure 5 Some visualizations of the prediction results were made, showing that this approach is superior to the other nine methods compared, especially in the recovery of the overall structure of salient objects. This approach strengthens the reasoning of the correspondence between pixels and supplements the boundary information lacking in graffiti annotations, thus enabling better prediction of complete salient objects.

[0147] Table 2 shows the performance of this approach compared to other methods on the DUTS-TE test set in terms of MAE and F. β Comparison of indicators.

[0148] Table 2

[0149]

[0150]

[0151] According to Table 2, it can be seen that this scheme achieved optimal accuracy on this dataset, significantly improving MAE by 0.008 points compared to the original scheme. β Significant improvements have been achieved.

[0152] Table 3 presents the performance of this approach compared to other methods on the DUT-OMRON test set in terms of MAE and F. β Comparison of indicators.

[0153] Table 3

[0154]

[0155] According to Table 3, it can be seen that this scheme achieves optimal accuracy on this dataset. Similarly, the MAE and F values ​​on this dataset are also optimal. β Significant improvements were achieved with accuracies of 0.068 and 0.720, respectively.

[0156] Table 4 shows the performance of this approach compared to other methods on the HKU-IS test set in terms of MAE and F. β Comparison of indicators.

[0157] Table 4

[0158]

[0159] Table 4 shows that this scheme achieves optimal accuracy on this dataset, with an MAE of 0.038 and F0.038 compared to the previous optimal method. β There is still some improvement even when it reaches 0.870. To clarify the role of the graph complementarity reasoning network on this dataset, Figure 5 Visualizations were performed, showcasing the prediction results in several complex scenarios. The results demonstrate that, even with weakly supervised training using graffiti annotations, this approach can accurately predict the overall structure of salient objects. This demonstrates that graph inference networks can infer long-range interactions between pixels, which is beneficial for the propagation of energy from annotated graffiti regions to unannotated regions during weakly supervised learning, thus leading to higher-quality predictions.

[0160] In some other embodiments, the above method is implemented as a salient object detection system. Exemplarily, the system is implemented by inputting an image to be segmented into a trained object detection model, thereby obtaining the output object segmentation result. The object detection model includes a base network module, a graph inference network module, a correction module, and a fusion module. The base network module is configured to extract multi-scale features from the sample image, thereby obtaining base edge features, base salient features, and coarse-grained salient features. The graph inference network module is configured to obtain corresponding graph representations based on the base edge features and base salient features. After cross-complementing the representations, the representation information is propagated using a graph convolutional network, thereby obtaining enhanced edge features and enhanced salient features. The correction module is configured to correct the enhanced edge features to obtain salient edge features. The fusion module is configured to fuse the salient edge features, enhanced salient features, and coarse-grained salient features to obtain the object segmentation result.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods or systems disclosed herein can be implemented using software plus necessary general-purpose hardware, or they can be implemented using dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. Generally, any function performed by a computer program can be easily implemented using corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the purposes of this disclosure, software implementation is often a preferred implementation method.

[0162] Although the embodiments of this invention have been described above in conjunction with the accompanying drawings, this invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of the claims of this invention, and all of these are within the scope of protection of this invention.

Claims

1. A graffiti-style weakly supervised salient object detection method based on complementary graph reasoning networks, characterized in that: The method inputs the image to be segmented into a trained object detection model and outputs the object segmentation result; wherein: the object detection model is weakly supervised training using graffiti annotation, and the steps include: Multi-scale features are extracted from sample images to obtain basic edge features, basic saliency features, and coarse-grained saliency features. The acquisition steps include: using a five-layer convolutional neural network to extract image features, and obtaining the features of each layer in sequence. (i=1,2,3,4,5); Low-level features and After resampling, the basic edge features are obtained by concatenation, and the high-level features are then... , and Basic saliency features are obtained by concatenating resampled data; high-level features are then used to further refine the data. Obtain coarse-grained significance features ; Based on basic edge features and basic saliency features, corresponding graph representations are obtained. After the representations are cross-complemented, the representation information is propagated using a graph convolutional network to obtain enhanced edge features and enhanced saliency features. The enhanced edge features are then modified to obtain salient edge features; The salient edge features, enhanced salient features, and coarse-grained salient features are fused together to obtain the target segmentation result; The training loss of the object detection model is composed of the gated conditional random field loss. Edge prediction loss saliency prediction loss for graffiti annotation pixel regions constitute.

2. The method according to claim 1, characterized in that, The steps to enhance edge features and enhance saliency features include: Based on fundamental edge features and fundamental saliency features, edge graph representations are obtained using graph projection. Significance plot representation ; Representing the edge map Significance plot representation Spatial location encoding is performed separately, and the edge map representation is obtained by cross-complementing the representations of the two using a multilayer perceptron model. Significance plot representation The representation information is then propagated through a graph convolutional network to obtain an enhanced edge graph representation. Enhanced saliency plot representation Then, backprojection is used to restore the original spatial coordinates of the two objects, which are used to enhance edge features. and enhanced saliency features .

3. The method according to claim 2, characterized in that, The crossover rules are as follows: Where: Relationship matrix It is obtained by calculating the correspondence between graph representations. and This represents a combination of 1D convolution, batch normalization, and the ReLU activation function. As the first characterization cross unit, As the second characterization cross unit, Indicates will After initial processing by the MLP layer, edge graph representation is performed before the intersection. Indicates will After initial processing by the MLP layer, a saliency graph representation is performed before the representation crossover, where D is the channel dimension. R represents the preset number of cluster centers in the graph projection, and R is the edge graph representation. Or saliency plot representation The number field.

4. The method according to claim 2, characterized in that, The method for calculating the propagation of representation information in graph convolutional networks is as follows: In the formula: For graph convolutional networks, It is a non-linear activation function, similarity matrix , , and It is a set of learnable parameters.

5. The method according to claim 2, characterized in that, Enhanced edge features and enhanced saliency features The acquisition steps include: Based on basic edge features Obtain the corresponding soft allocation matrix Based on fundamental saliency features Obtain the corresponding soft allocation matrix. ; The original spatial coordinate features are obtained by performing back projection calculations using the following formula: in: It is the feature dimension space of the original spatial coordinates, where H is the height of the edge feature or salient feature, W is the width of the edge feature or salient feature, and D represents the channel dimension of the edge feature or salient feature.

6. The method according to claim 1, characterized in that, The steps for obtaining salient edge features include: Will enhance edge features Features are obtained through convolution processing and calculate features Channel weights This leads to the obtained corrected edge features. : In the formula: and Represents a convolutional layer. Indicates global average pooling. It is a non-linear activation function. This represents the sigmoid function. Indicates pixel-by-pixel multiplication; Use corrected edge features Obtain enhanced edge features It is then connected to the edge features extracted from the input image using the Canny operator, and then the number of channels is adjusted using convolutional layers to generate salient edge features. ,Right now: In the formula: It is the Canny edge extraction operator. I Indicates the input image. This is for channel-based splicing operations. This represents adding pixels one by one.

7. The method according to claim 1, characterized in that, The steps to achieve integration include: Based on enhanced saliency features Channel adjustments were made to obtain enhanced significance features. ; This will enhance saliency features. Significant edge features and coarse-grained saliency features The calculation rules for the dense fusion strategy are as follows: In the formula: Represents a convolutional layer. Up Indicates an upsampling operation. Indicates pixel-by-pixel multiplication. This indicates a channel splicing operation.

8. The method according to claim 1, characterized in that, The training loss of the object detection model is calculated as follows: In the formula: This is a partial cross-entropy function. For partial cross-entropy loss, For edge loss; , To predict the significance score of pixel i in the image, To predict the significance score of pixel j in the image, To predict the difference in significance scores between pixel i and pixel j in the image, It is the window size for the energy diffusion of pixel i. It is a similarity energy function; where the significance score is obtained by regressing the fusion result with the graffiti label, utilizing... softmax The function transforms the regression results to obtain the desired outcome. The pixel area marked for the graffiti. It is the edge truth region. For edge prediction region, For the region of significance, This represents the region of significant prediction.

9. A salient target detection system, characterized in that: The system inputs the image to be segmented into a trained target detection model and outputs the target segmentation result. The target detection model includes a basic network module, a graph inference network module, a correction module, and a fusion module; The basic network module is configured to extract multi-scale features from sample images, thereby obtaining basic edge features, basic saliency features, and coarse-grained saliency features. The acquisition steps include: using a five-layer convolutional neural network to extract image features, and sequentially obtaining the features of each layer. (i=1,2,3,4,5); Low-level features and After resampling, the basic edge features are obtained by concatenation, and the high-level features are then... , and Basic saliency features are obtained by concatenating resampled data; high-level features are then used to further refine the data. Obtain coarse-grained significance features ; The graph reasoning network module is configured to obtain corresponding graph representations based on basic edge features and basic saliency features. After the representations are cross-complemented, the representation information is propagated using a graph convolutional network to obtain enhanced edge features and enhanced saliency features. The correction module is configured to correct the enhanced edge features to obtain salient edge features; The fusion module is configured to fuse salient edge features, enhanced salient features, and coarse-grained salient features to obtain the target segmentation result; Wherein: the training loss of the object detection model is composed of the gated conditional random field loss. Edge prediction loss saliency prediction loss for graffiti annotation pixel regions constitute.

Images