A weakly supervised salient object detection method based on enhanced scribble annotation

Abstract

The application discloses a weakly supervised salient object detection method based on enhanced scribble annotation, comprising: a label expansion method, which expands original labels on a disclosed scribble dataset S-DUTS, and inputs the expanded labels into a self-adaptive network SANet framework for training; through mining of potential semantic information in high-level features and structural information contained in low-level features and an information interaction fusion module, effective fusion between different hierarchical features is realized; meanwhile, in the training process, an adaptive boundary alignment module is combined with an adaptive gating loss, so that the model can perceive the optimal boundary of the target without introducing other auxiliary information. According to the application, compared with the most advanced weakly supervised and unsupervised methods, competitive performance is achieved on five benchmark datasets, and the performance of some fully supervised methods is exceeded; more effective scribble annotations are obtained without increasing the labor cost, cross-scale edge information is enhanced through an adaptive boundary alignment ABA module, and more detailed information is learned without supplementing edge data.

Description

Technical Field

[0001] This invention relates to the technical field of object detection, and in particular to a weakly supervised saliency object detection method based on enhanced graffiti annotation. Background Technology

[0002] Salient object detection aims to mimic the complex attention mechanisms of human vision to identify and detect the most visually salient objects in a scene. As an effective preprocessing technique, it has wide applications in various downstream vision tasks such as medical inspection, human-computer interaction, and object tracking. Early traditional salient object detection methods mainly relied on handcrafted features (such as texture, color, and contrast) to distinguish foreground salient regions from the background. However, these handcrafted features cannot extract high-level semantic information for accurate salient segmentation, especially in complex scenes. In recent years, the superior feature capture capabilities of CNNs have replaced handcrafted features and achieved extremely high performance.

[0003] While many methods have achieved excellent results, their effectiveness heavily relies on a large number of accurate pixel-level labels, a process that requires significant manual and time investment. Therefore, it is necessary to find weakly supervised methods to alleviate the annotation overhead. Existing weakly supervised salient object detection (SOD) annotation methods include: image-level annotation, pseudo-labels, bounding boxes, doodle annotations, and point annotations. Early weakly supervised SOD methods, inspired by classification networks, proposed image-level annotations to replace pixel-level annotations. However, due to the lack of precise location information in these labels, the generated saliency maps are incomplete. Subsequently, models trained using pseudo-labels were proposed, achieving some performance improvements. These labels are obtained through manual methods or prediction maps generated by other weakly supervised / unsupervised models. However, these methods typically require complex training strategies or iterative mechanisms to obtain effective saliency maps. In recent years, methods such as bounding boxes, doodle annotations, and point annotations have emerged, not only reducing the time required for manual annotation but also providing some location and structural information. However, these methods still rely on additional data information. Compared to other weakly supervised methods (e.g., image-level, bounding box, and point-level), graffiti annotations provide more pixel-level guidance by leveraging semantic information and are relatively more effective in labeling. However, while saving costs (e.g., manpower and time) using graffiti annotation labels, a significant loss of accuracy is also incurred. Due to the limited number of annotated pixels in graffiti supervision, the lack of semantic and structural information inhibits the model's ability to acquire the necessary information, resulting in a substantial performance gap compared to pixel-level labels. In conclusion, obtaining a complete saliency map in weakly supervised salient object detection remains a formidable challenge. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a weakly supervised salient object detection method based on enhanced graffiti annotation. This method achieves competitive performance with state-of-the-art weakly supervised and unsupervised methods on five benchmark datasets, surpassing the performance of some fully supervised methods. It obtains more effective graffiti annotations without increasing manual costs, enhances cross-scale edge information through an adaptive boundary alignment (ABA) module, and learns more detailed information without requiring supplementary edge data. To achieve the above-mentioned objectives and other advantages of the present invention, a weakly supervised salient object detection method based on enhanced graffiti annotation is provided, comprising:

[0005] The label expansion method expands the original labels on the publicly available graffiti dataset S-DUTS, and then inputs the expanded labels into the adaptive network SANet framework for training. By mining the potential semantic information in high-level features and the structural information contained in low-level features, as well as the information interaction fusion module, it achieves effective fusion between features at different levels.

[0006] Meanwhile, during training, the adaptive boundary alignment module combined with adaptive gating loss enables the model to perceive the optimal boundary of the target without introducing other auxiliary information.

[0007] Specifically, the tag expansion method includes the following steps:

[0008] S1. Perform superpixel segmentation on the original image, and map the superpixel segments of the original image to the background label S. b and foreground label S f On top, obtain the superpixels of the label;

[0009] S2. If the proportion of labeled pixels reaches a certain level, which is: set a diffusion threshold. If the proportion of labeled pixels mapped in a superpixel is greater than the threshold, the entire superpixel will be labeled with the same label.

[0010] S3. Taking a superpixel block of the background label as an example, the number of marked pixels and the total number of pixels in the superpixel block are respectively n. l and n, only when n l / n>T b Only all pixels of the superpixel are marked as background; otherwise, no expansion is performed.

[0011] Because the background pixels account for a large proportion, two expansions were implemented, with the number of superpixel segments and the diffusion threshold (N) being adjusted for each expansion. b T b Let (500, 0.1) and (400, 0.2) be set respectively;

[0012] Since the foreground pixels occupy a smaller proportion, an expansion is set once, where the number of segmentation blocks and the threshold (N) are adjusted. f T f Let (400, 0.2) be the integer part of the integer part.

[0013] Preferably, the adaptive network SANet framework adopts an encoder-decoder architecture, which includes a ResNet50 encoder, a global semantic interaction (GSA) module, a local structure enhancement (LSE) module, and an adaptive boundary alignment (ABA) module. The specific operation of the adaptive network SANet framework includes the following steps:

[0014] The dataset images are input into the ResNet50 backbone network for image encoding, resulting in five-layer multi-scale feature maps f. i (i = 0, 1, 2, 3, 4);

[0015] High-level features (i.e., the fourth and fifth layer feature maps) are input into the Global Semantic Interaction (GSA) module to learn global semantic information, while low-level features (i.e., the second and third layer feature maps) are input into the Local Structure Enhancement (LSE) module to capture local structure and contextual information.

[0016] A global pyramid pooling layer is added to the fifth layer of features to further extract global semantic information. The high-level features extracted by the global pyramid pooling layer and the global semantic interaction GSA module are fused separately.

[0017] The enhanced high-level features obtained after fusion are then cross-feature aggregated with the low-level features extracted by the Local Structure Enhancement (LSE) module.

[0018] The final two layers of features are fused, and features are extracted through a 1×1 convolution, batch normalization, and ReLU function. The final prediction map is obtained by the sigmoid activation function.

[0019] Preferably, the Global Semantic Interaction (GSA) module combines four convolutional kernels with different target feature capture capabilities: asymmetric convolution, dilated convolution, grouped convolution, and ordinary convolution. The features obtained after convolution with these four kernels are concatenated as follows:

[0020]

[0021] In the above formula, The GSA module represents the features extracted by the Global Semantic Interaction (GSA) module. Conv3BR(·) represents a 3×3 convolutional layer after batch normalization and ReLU operation. Cat[·] represents channel-level concatenation. asyConv3(·) represents a 3×3 asymmetric convolution. groConv3(·) is a 3×3 dilated convolution with a dilation rate of 3, groConv3(·) is a 3×3 grouped convolution with GELU activation and batch normalization, and Conv5(·) represents a 5×5 convolution.

[0022] Preferably, the global features further extracted through the global pyramid pooling layer reduce the loss of contextual information. The global pyramid pooling layer and the high-level features extracted by the global semantic interaction (GSA) module are fused to obtain two intermediate feature maps, as shown in the following equation:

[0023]

[0024] In the above formula, For GPP, For feature fusion; feature fusion is performed on two input features (f) (1) f (2) Element-wise multiplication and element-wise summation are performed between the elements; subsequently, feature fusion is represented sequentially by two 3×3 convolutions, batch normalization, and ReLU activation, as shown in the following equation:

[0025]

[0026] In the above formula, It is element-wise multiplication. It is an element-wise summation.

[0027] Preferably, the Local Structure Enhancement (LSE) module is used to enhance and extract detailed information and local contextual cues, accurately detecting the structure of salient target regions. This LSE module includes two dual-spatial attention DSA modules with different receptive fields. After batch normalization and ReLU activation, the outputs of the two dual-spatial attention DSA modules are concatenated. One of the dual-spatial attention DSA modules contains a stack of two coordinate attention mechanisms, as shown in the following equation:

[0028]

[0029] In the above formula, Conv n (·) represents an n×n convolution, CA(·) is the output of the coordinate annotation, and BR(·) is the batch normalization and ReLU activation operation.

[0030] Preferably, the Local Structure Enhancement (LSE) module consists of a 3×3 convolution and two parallel dual-spatial attention (DSA) modules. In each LSE module, besides the dilated convolution... The void ratio 'a' is set to different values, 2 and 4 respectively, while the rest remain the same. The feature output of the Local Structure Enhancement (LSE) module is shown in the following formula:

[0031]

[0032] In the above formula, This represents the feature output after passing through the Local Structure Enhancement (LSE) module. This feature is then cross-aggregated with the output features of the Global Semantic Interaction (GSA) module to obtain the corresponding intermediate feature map.

[0033] Preferably, the output features of the Local Structure Enhancement (LSE) module are cross-feature aggregated with the global semantic information obtained from the high-level features to obtain two intermediate feature maps, as shown in the following equation:

[0034]

[0035] In the above formula, The cross-aggregation module is represented as shown in the following formula:

[0036]

[0037] In the above formula, f l It is an enhancement feature based on the structural information of local structure enhancement LSE, f g Up is an enhanced high-level feature based on global semantic information. ×2 This represents two upsampling operations; the two intermediate feature maps are then fused again, as shown in the following formula:

[0038]

[0039] In the above formula, feature F0 is then processed by a 1×1 convolution, batch normalization, and ReLU function to extract features, and the sigmoid activation function is used to obtain the final prediction map.

[0040] Preferably, the adaptive boundary alignment ABA module is used to extract boundaries without requiring additional edge data supervision, avoiding the introduction of other noise information. Furthermore, this adaptive boundary alignment ABA module gradually extracts boundary details from the U-shaped network through a boundary extractor. And combine multi-scale salient target information The reuse of target information in the boundary extractor ensures the structural integrity of the target boundary and shields it from interference from insignificant background.

[0041] Boundary feature B i+1 It can be enhanced by feature F i+1 The fusion method to enhance boundary features can be written as:

[0042]

[0043] Accordingly, the entire process of the ABA module can be formally expressed as follows:

[0044]

[0045] Among them, E i (·) represents the i-th boundary extractor, specifically extracting features from the previous layer. With the current enhanced feature F i The components are connected together, and then three convolution operations are performed. Boundary features B are extracted through average pooling, element-wise subtraction, and PReLU activation. i The corresponding expression is as follows:

[0046]

[0047]

[0048] Where AP(·) represents the average pooling operation, and PR(·) represents the Prelu activation operation.

[0049] Preferably, during training, the adaptive gating loss function and its internal gating mechanism guide the model to perceive the optimal boundary of salient targets. In the initial stage of training, the adaptive gating loss corrects the errors between the boundaries of low-level feature extraction in a self-aligning manner. Through the generation of the prediction map, the structural information of the salient targets is activated at a certain gate, and its guiding weights are adaptively adjusted. The structural information of the salient targets is supervised by an adaptive quantization boundary Q, as described below:

[0050]

[0051] In the above formula, I gradient It is the gradient map of the input image. This is a pixel-level multiplication, where P is the predicted saliency map; the adaptive gated loss is implemented using mean squared error loss, as described below:

[0052]

[0053] In the above formula, B1 and B2 are the boundary features of the low-level features extracted by the adaptive boundary alignment ABA module, and ω g The auxiliary guidance used to balance structural information is described as follows:

[0054]

[0055] In the above formula, t represents the current training period. d β and β are hyperparameters, set to 40 and 0.25 respectively; finally, the adaptive gating loss, along with the local significant coherence loss, significant structural consistency loss, and partial cross-entropy loss, jointly guide the model training.

[0056] Compared with existing technologies, the advantages of this invention are as follows: First, it designs a comprehensive framework suitable for weakly supervised salient object detection. Under the supervision of graffiti annotations, it utilizes the latent semantic information in high-level features and the structural information contained in low-level features, achieving effective extraction of features at different levels through information interaction and fusion. Second, addressing the problem of weakly supervised salient object detection, it proposes an adaptive boundary alignment module. This module enables the network to learn object boundary information without relying on additional data or complex training procedures, thereby generating high-quality salient maps. Simultaneously, the boundary self-alignment mechanism in the adaptive boundary alignment module, combined with the adaptive gating loss function proposed in this invention, can adaptively focus on the boundary regions of salient objects, guiding the model to effectively perceive the boundaries of salient objects.

[0057] The method of this invention achieves competitive performance on multiple public benchmark datasets, outperforming other weakly supervised methods for graffiti annotation and even some fully supervised methods. It generates more accurate saliency maps in various scenarios, including cluttered backgrounds, large objects, small objects, multiple objects, and low-contrast scenes. Attached Figure Description

[0058] Figure 1 This is an adaptive network framework diagram of the weakly supervised saliency target detection method based on enhanced graffiti annotation according to the present invention;

[0059] Figure 2 This is a schematic diagram of the graffiti annotation expansion method according to the present invention, which is based on the weakly supervised salience target detection method with enhanced graffiti annotation.

[0060] Figure 3 This is a schematic diagram of the pseudo-label after graffiti annotation expansion in the weakly supervised salience target detection method based on enhanced graffiti annotation according to the present invention;

[0061] Figure 4 This is a schematic diagram of the dual-space attention DSA module structure in the Local Structure Enhancement (LSE) module of the weakly supervised saliency target detection method based on enhanced graffiti annotation according to the present invention.

[0062] Figure 5 This is a schematic diagram of the boundary extractor structure in the adaptive boundary alignment ABA module of the weakly supervised saliency target detection method based on enhanced graffiti annotation according to the present invention;

[0063] Figure 6 This is a visualization of the weakly supervised saliency target detection method based on enhanced graffiti annotation according to the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] To address the label sparsity problem, this application first preprocesses the labels using superpixels, achieving more effective labels without incurring additional manual costs. To overcome the limitations imposed by label sparsity, a global semantic association module is introduced into the high-level features with semantic information. This module utilizes four different convolutional kernels to capture different target features. The features obtained after convolution with these kernels are combined to establish global semantic interaction, helping to explore additional latent semantic information to enrich and refine the salient objects. For low-level features with structural information, a local structure enhancement module is proposed, aiming to enhance and mine the structural features of potentially salient targets, helping the graffiti spread to potentially salient target regions. Furthermore, a boundary self-alignment module is designed; compared to other boundary alignment methods, we do not need to introduce additional edge information. By incorporating an adaptive boundary alignment mechanism into the feature extraction of adjacent layers and combining it with the proposed adaptive gating loss, we adaptively focus on the boundaries of indistinguishable objects.

[0066] like Figure 1 As shown, this invention proposes an adaptive network framework for a weakly supervised salient object detection method based on enhanced graffiti annotation. This framework can obtain a relatively complete salient object prediction map without introducing any additional information. First, to address the sparsity of labels, a label expansion method is used to expand the original labels on the publicly available graffiti dataset S-DUTS. Second, the input image is encoded through a ResNet50 backbone network to obtain a five-layer multi-scale feature map f. i (i = 0, 1, 2, 3, 4) Semantic information is extracted from high-level features (f3 and f4) through the Global Semantic Association (GSA) module, and salient target structural information is mined from low-level features (f1 and f2) through the Local Structure Enhancement (LSE) module. Furthermore, to avoid interference from background noise introduced by extra information, an adaptive boundary alignment (ABA) module that does not require additional edge information is designed. Through the boundary extractor and boundary self-alignment mechanism in this module, combined with the proposed adaptive gating loss, the model can adaptively cluster to guide the perception of salient target boundaries.

[0067] like Figure 2The diagram illustrates the graffiti annotation expansion method of this invention. Graffiti annotations only label a small portion of the foreground and background; unlabeled pixels are considered unknown. Due to the sparsity of its labels, this model cannot perceive and recognize objects with complex shapes and blurred boundaries. Unlike other graffiti expansion methods that require additional edge maps, this method automatically segments regions based on image information (color, texture, brightness, etc.) using a superpixel algorithm to expand labels. This superpixel algorithm is existing technology and will not be described in detail here. The superpixel algorithm, by grouping adjacent pixels with similar colors, brightness, and textures into a segment, has advantages such as fast computation speed, high memory efficiency, and ease of use. First, superpixel segmentation is performed on the original image; then, the superpixel segmentation of the original image is mapped to the background label S. b and foreground label S f The superpixels of the labeled image are obtained. For each superpixel block of the labeled image, if the proportion of labeled pixels reaches a certain level, which is defined as: a diffusion threshold is set, and if the proportion of labeled pixels mapped within a certain superpixel is greater than this threshold, the entire superpixel will be labeled with the same label. Considering the imbalance between background pixels and salient targets, different diffusion thresholds are set for foreground and background labels when labeling superpixels.

[0068] like Figure 3 The diagram shows a pseudo-label after the graffiti annotation has been expanded. The expanded graffiti annotation can capture more valuable information, especially in the case of complex objects and complex boundaries, providing more supervision information to the network and obtaining a high-quality final prediction graph.

[0069] Secondly, the Global Semantic Interaction (GSA) module incorporates four convolutional kernels with different target feature capture capabilities: asymmetric convolution, dilated convolution, grouped convolution, and ordinary convolution. The features obtained after convolution with these kernels are combined to establish global semantic interaction, helping to explore potential semantic information to enrich and extract salient targets. This process can be described as follows:

[0070]

[0071] in, The GSA module represents the features extracted by the Global Semantic Interaction (GSA) module. Conv3BR(·) represents a 3×3 convolutional layer after batch normalization and ReLU operation. Cat[·] represents channel-level concatenation. asyConv3(·) represents a 3×3 asymmetric convolution. `groConv3(·)` is a 3×3 dilated convolution with a dilation rate of 3, `groConv3(·)` is a 3×3 grouped convolution with GELU activation and batch normalization, and `Conv5(·)` represents a 5×5 convolution. Compared to ordinary convolutions, asymmetric convolutions can enhance feature representation by learning from the central kernel, natural convolutions can expand the receptive field and capture multi-scale contextual semantic information, and group convolutions enhance the model's representational ability by introducing interactions between groups. By fusing features learned from these kernels, the network benefits from a rich receptive field and diverse feature representations. This approach helps to establish global semantic connections, thereby exploring the integrity of underlying semantic information and salient objects.

[0072] Meanwhile, to reduce the loss of contextual information, global semantic features are further extracted from the top-level features through a global pyramid pooling layer, and combined with the high-level features extracted by the Global Semantic Interaction (GSA) module to obtain two intermediate feature maps. This process can be described as follows:

[0073]

[0074] in, For GPP, For feature fusion; feature fusion is performed on two input features (f) (1) f (2) Element-wise multiplication and element-wise summation are performed between them. Subsequently, feature fusion is represented sequentially by two 3×3 convolutions, batch normalization, and ReLU activation, as described in (3):

[0075] In the above formula It is element-wise multiplication. It is an element-wise summation.

[0076] Low-level features contain rich texture and detail information, effectively emphasizing the structural properties and contours of salient objects. However, in doodle annotations, the sparsity of labeled pixels makes extracting the structure of salient targets difficult. To address this issue, a Local Structure Enhancement (LSE) module is introduced into the two low-level feature layers. This module aims to enhance and extract fine-grained feature information and local contextual cues, accurately detecting salient target regions. By embedding positional information into channel attention, coordinate attention can capture not only cross-channel information but also orientation and positional awareness. The LSE module utilizes coordinate attention to enhance the model's feature representation capabilities, helping the model to more accurately locate and identify targets. Each LSE module includes two dual-spatial attention (DSA) modules with different receptive domains.

[0077] like Figure 4The diagram shows the structure of the dual-space attention DSA module in the Local Structure Enhancement (LSE) module. The architecture of a dual-space attention DSA module comprises a stack of two coordinate attention mechanisms. This process can be described as follows:

[0078]

[0079] Among them, Conv n (·) denotes an n×n convolution, CA(·) is the output of coordinate attention, and BR(·) is the batch normalization and ReLU activation operation. The Local Structure Enhancement (LSE) module consists of a 3×3 convolution and two parallel dual-spatial attention (DSA) modules. In each LSE module, besides the dilated convolution... The void ratio 'a' is set to different values, 2 and 4 respectively, while the rest remain the same. The feature output of the Local Structure Enhancement (LSE) module can be described as follows:

[0080]

[0081] in, This represents the feature output after the Local Structure Enhancement (LSE) module. This feature is then cross-aggregated with global semantic information obtained from high-level features to obtain the corresponding intermediate feature map. This process can be described as follows:

[0082]

[0083] in, This represents the cross-aggregation module. The process can be described as follows:

[0084]

[0085] Among them, f l It is an enhancement feature based on the structural information of local structure enhancement LSE, f g Up is an enhanced high-level feature based on global semantic information. ×2 This represents two upsampling operations; the two intermediate feature maps are then fused again. This process can be described as follows:

[0086]

[0087] In this process, feature F0 is further processed by a 1×1 convolution, batch normalization, and ReLU function to extract features, and the sigmoid activation function is used to obtain the final prediction map.

[0088] The adaptive boundary alignment ABA module proposed in this invention can extract boundaries without the need for additional edge maps, thereby avoiding the introduction of noise and reducing computational costs. Simultaneously, a U-shaped structure is constructed to iteratively refine the boundary details of salient targets, enabling it to focus on boundaries while ensuring the integrity and accuracy of salientity inference. Unlike existing models that only use some low-level and high-level deep features to mine boundary features, the adaptive boundary alignment ABA module aims to progressively mine boundary details from multi-scale features.

[0089] Specifically, the ABA module contains four boundary extractors. For the i-th boundary extractor, its input is the enhanced feature F from the current layer. i And boundary features from the upper layer. In other words, boundary feature B i+1 It can be enhanced by feature F i+1 The fusion method to enhance boundary features can be written as:

[0090]

[0091] in, This is the enhanced boundary feature. The reuse of object information not only ensures the structural integrity of the object boundary but also prevents interference from insignificant backgrounds, thus effectively enhancing the boundary feature.

[0092] Accordingly, the entire process of the ABA module can be formally expressed as follows:

[0093]

[0094] Among them, E i (·) represents the i-th boundary extractor, whose structure is as follows: Figure 5 As shown. First, the features from the previous layer... With the current enhanced feature F i The components are connected together, and then three convolution operations are performed. Boundary features B are extracted through average pooling, element-wise subtraction, and PReLU activation. i The corresponding expression is as follows:

[0095]

[0096]

[0097] Where AP(·) represents the average pooling operation, and PR(·) represents the Prelu activation operation.

[0098] Furthermore, considering the quality of self-generated boundary information, a loss function with a gating supervision strategy and a self-alignment mechanism is designed to optimize the boundary mapping of low-level features, providing sufficient guidance for obtaining the saliency target of complete boundaries.

[0099] Due to the sparsity of graffiti annotations, sufficient supervision cannot be provided for pixels on object boundaries; and in the early stages of training, the detected salient targets are not reliable enough to provide reasonable supervision; in order to mine the potential structural information of salient targets and provide auxiliary guidance for the network, adaptive gating loss corrects the error between the boundaries of low-level feature extraction in a self-aligning manner in the initial stage of training; as reliable prediction maps are generated, the structural information of salient targets is activated in a certain gate and adaptively adjusts its guiding weight; where the structural information of salient targets is supervised by an adaptive quantized boundary Q-supervised adaptive boundary alignment module, expressed as (12):

[0100]

[0101] In the above formula, I gradient It is the gradient map of the input image. For pixel-level multiplication, P is the predicted saliency map; the adaptive gated loss is achieved using mean squared error loss, expressed as (13):

[0102]

[0103] In the above formula, B1 and B2 are the boundary features of the low-level features extracted by the adaptive boundary alignment ABA module, and ω g The auxiliary guidance used to balance structural information is described as (14):

[0104]

[0105] In the above formula, t represents the current training period. d β and α are hyperparameters, set to 40 and 0.25 respectively. In the early stages of training, MSE loss guides the two sets of low-level features to correct boundary errors in a self-aligning manner. As training progresses, the predicted salient targets gradually become more reliable. The structural information of the salient targets guides the activation of a certain gate, which gradually strengthens as training continues.

[0106] Finally, the adaptive gating loss, along with the local salient coherence loss, salient structural consistency loss, and partial cross-entropy loss, jointly guides the model training; ultimately, the trained model can be directly used for salient object detection.

[0107] Figure 6 Visualization examples of the results of this invention are provided. The first column is the original color image, the second column is the ground truth map of salient object detection, the third column is the saliency map predicted by the latest weakly supervised model, and the fourth column is the saliency map predicted by this invention. The results show that the method of this invention achieves better visual results in many challenging scenarios.

[0108] The number of devices and processing scale described herein are for the purpose of simplifying the description of the invention, and applications, modifications and variations thereof will be apparent to those skilled in the art.

[0109] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A weakly supervised saliency target detection method based on enhanced graffiti annotation, characterized in that, The label expansion method expands the original labels on the publicly available graffiti dataset S-DUTS, and then inputs the expanded labels into the adaptive network SANet framework for training. By mining the potential semantic information in high-level features and the structural information contained in low-level features, as well as the information interaction fusion module, it achieves effective fusion between features at different levels. Meanwhile, during training, the adaptive boundary alignment module combined with adaptive gating loss enables the model to perceive the optimal boundary of the target without introducing other auxiliary information. Specifically, the tag expansion method includes the following steps: S1. Perform superpixel segmentation on the original image, and map the superpixel segments of the original image to the background labels. and foreground tags On top, obtain the superpixels of the label; S2. If the proportion of labeled pixels reaches a certain level, which is: set a diffusion threshold. If the proportion of labeled pixels mapped in a superpixel is greater than the threshold, the entire superpixel will be labeled with the same label. S3. For a superpixel block of the background label, the number of marked pixels and the total number of pixels in the superpixel block are respectively... and Only when Only all pixels of the superpixel are marked as background; otherwise, no expansion is performed. Because the background pixels account for a large proportion, two expansions were implemented, with the number of superpixel segments and the diffusion threshold adjusted for each expansion. Set as follows and ; Since the foreground pixels occupy a smaller proportion, an expansion is set once, involving the number of segmentation blocks and the threshold. Set as ; The adaptive network SANet framework adopts an encoder-decoder architecture, which includes a ResNet50 encoder, a Global Semantic Interaction (GSA) module, a Local Structural Enhancement (LSE) module, and an Adaptive Boundary Alignment (ABA) module. The SANet framework operates by performing the following steps: The dataset images are input into the ResNet50 backbone network to encode the images, resulting in five layers of multi-scale feature maps. ; High-level features, namely the fourth and fifth layer feature maps, are input into the Global Semantic Interaction (GSA) module to learn global semantic information, while low-level features, namely the second and third layer feature maps, are input into the Local Structure Enhancement (LSE) module to capture local structure and contextual information. A global pyramid pooling layer is added to the fifth layer of features to further extract global semantic information. The high-level features extracted by the global pyramid pooling layer and the global semantic interaction GSA module are fused separately. The enhanced high-level features obtained after fusion are then cross-feature aggregated with the low-level features extracted by the Local Structure Enhancement (LSE) module. The final two layers of features are fused, and features are extracted through a 1×1 convolution, batch normalization, and ReLU function. The final prediction map is obtained by the sigmoid activation function.

2. The weakly supervised saliency target detection method based on enhanced graffiti annotation as described in claim 1, characterized in that, The Global Semantic Interaction (GSA) module combines four convolutional kernels with different target feature capture capabilities: asymmetric convolution, dilated convolution, grouped convolution, and ordinary convolution. The features obtained after convolution with these four kernels are concatenated as follows: （1） In the above formula, This represents the features extracted by the Global Semantic Interaction (GSA) module. This represents a value after batch normalization and ReLU operation. Convolutional layer This indicates channel-level splicing. Represent a Asymmetric convolution, It has a void ratio of 3. Hollow convolution, It is Grouped convolution with GELU activation and batch normalization. express convolution.

3. The weakly supervised saliency target detection method based on enhanced graffiti annotation as described in claim 2, characterized in that, The global features extracted by the global pyramid pooling layer reduce the loss of contextual information. The global pyramid pooling layer and the high-level features extracted by the global semantic interaction (GSA) module are fused to obtain two intermediate feature maps, as shown in the following equation: （2） In the above formula, For GPP, For feature fusion; feature Fusion is performed on two input features respectively. Element-wise multiplication and element-wise summation are performed between them; subsequently, feature fusion is represented sequentially by two 3×3 convolutions, batch normalization, and ReLU activation, as shown in the following equation: （3） In the above formula, It is element-wise multiplication. It is an element-wise summation.

4. The weakly supervised saliency target detection method based on enhanced graffiti annotation as described in claim 1, characterized in that, The Local Structure Enhancement (LSE) module is used to enhance and extract detailed information and local contextual cues, accurately detecting the structure of salient target regions. This LSE module includes two dual-spatial attention DSA modules with different receptive fields. After batch normalization and ReLU activation, the outputs of the two dual-spatial attention DSA modules are concatenated. One of the dual-spatial attention DSA modules contains a stack of two coordinate attention mechanisms, as shown in the following equation: （4） In the above formula, express convolution, For the output of coordinate attention, It involves batch normalization and ReLU activation operations.

5. The weakly supervised saliency target detection method based on enhanced graffiti annotation as described in claim 4, characterized in that, The Local Structure Enhancement (LSE) module consists of a 3×3 convolution and two parallel dual-spatial attention (DSA) modules; in each LSE module, in addition to dilated convolutions... Hollowness ratio The values ​​are different, set to 2 and 4 respectively, while the rest are the same. The feature output of the Local Structure Enhancement (LSE) module is shown in the following formula: （5） In the above formula, This represents the feature output after passing through the Local Structure Enhancement (LSE) module. This feature is then cross-aggregated with the output features of the Global Semantic Interaction (GSA) module to obtain the corresponding intermediate feature map.

6. The weakly supervised saliency target detection method based on enhanced graffiti annotation as described in claim 5, characterized in that, The output features of the Local Structure Enhancement (LSE) module are cross-feature aggregated with the global semantic information obtained from the high-level features to obtain two intermediate feature maps, as shown in the following equation: （6） In the above formula, The cross-aggregation module is represented as shown in the following formula: （7） In the above formula, It is an enhancement feature based on the structural information of LSE (Local Structure Enhancement). It is an enhanced high-level feature based on global semantic information. This represents two upsampling operations; the two intermediate feature maps are then fused again, as shown in the following formula: （8） In the above formula, the characteristics After a 1×1 convolution, batch normalization, and ReLU function to extract features, the sigmoid activation function is used to obtain the final prediction map.

7. The weakly supervised saliency target detection method based on enhanced graffiti annotation as described in claim 5, characterized in that, The adaptive boundary alignment ABA module is used to extract boundaries without requiring additional edge data supervision, avoiding the introduction of other noise information. Furthermore, this adaptive boundary alignment ABA module progressively extracts boundary details from the U-shaped network through a boundary extractor. And combine multi-scale salient target information The reuse of target information in the boundary extractor ensures the structural integrity of the target boundary and shields it from interference from insignificant background. Boundary features It can be enhanced by features The fusion method to enhance boundary features can be written as: （9） Accordingly, the entire process of the ABA module can be formally expressed as follows: （10） in, For the first Each boundary extractor specifically extracts features from the previous layer. Compared with the current enhanced features The components are connected together, and then three convolution operations are performed. Boundary features are extracted through average pooling, element-wise subtraction, and PReLU activation. The corresponding expression is as follows: （11） （12） in, For average pooling operation, This indicates the Prelu activation operation.

8. The weakly supervised saliency target detection method based on enhanced graffiti annotation as described in claim 7, characterized in that, During training, the adaptive gating loss function and its internal gating mechanism guide the model to perceive the optimal boundary of salient targets. In the initial stage of training, the adaptive gating loss corrects errors between low-level feature extraction boundaries in a self-aligning manner. Through the generation of the prediction map, the structural information of salient targets is activated at a certain gate, and its guiding weights are adaptively adjusted. The structural information of salient targets is determined by an adaptively quantized boundary. Supervision, as described below: （13） In the above formula, It is the gradient map of the input image. Pixel-level multiplication, The saliency map is the prediction map; the adaptive gated loss is implemented using the mean squared error loss, as described below: （14） In the above formula, and Boundary features of low-level features extracted by the adaptive boundary alignment ABA module. The auxiliary guidance used to balance structural information is described as follows: （15） In the above formula, For the current training cycle, and The hyperparameters were set to 40 and 0.25, respectively. Finally, the adaptive gating loss, along with the local significant coherence loss, significant structural consistency loss, and partial cross-entropy loss, jointly guided the model training.

Images