A method and system for detecting salient objects by integrating edge features

Abstract

The present invention provides a method and system for detecting a salient target by fusing edge features, wherein the method comprises: using a backbone network to process an image to obtain edge features and multiple groups of salient features; using a self-attention group pixel fusion module to process edge features and salient features to obtain salient edge features and salient target features; using a pooling model to process salient features to obtain global salient features; using a bidirectional feature fusion module to fuse salient edge features, salient target features and global salient features to obtain detection results. The method and system for detecting a salient target by fusing edge features of the present invention are specifically used for salient target detection, have the ability to self-optimize information within a layer, and improve the prediction accuracy of salient target detection tasks by effectively utilizing the complementarity between salient edge information and salient target information.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a salient object detection method and system integrating edge features. Background Art

[0002] Salient Object Detection aims to simulate the human visual attention mechanism, identify the most salient areas in the image, and accurately locate key foreground information. As a preprocessing step in the field of computer vision, this task has been widely used in multiple visual tasks such as image retrieval, visual tracking, image restoration, and photo synthesis.

[0003] Traditional salient object detection models are mainly based on bottom-up methods, using different underlying visual features to classify samples as salient or insignificant. Convolutional neural networks (CNNs) are widely used in different image tasks, such as object detection, semantic segmentation, edge detection, etc. For salient object detection (SOD), CNN has brought new ideas and has shown significant improvements in some studies. Due to the multi-stage and multi-scale characteristics of CNN, it can accurately capture the most salient areas without prior knowledge. In addition, even in the face of shadows or reflections, multi-stage features help CNN to more accurately locate the boundaries of salient objects. Due to these advantages, CNN-based SOD methods have refreshed historical records on almost all existing datasets and become mainstream methods.

[0004] Especially after the emergence of fully convolutional neural networks (FCNs), salient object detection (SOD) based on deep learning can better play its advantages and achieve effective feature expression. Inspired by FCNs, more and more pixel-level SOD methods have emerged.

[0005] Salient Object Detection (SOD) is an important task in the field of computer vision. Its goal is to accurately locate and segment the most salient objects or regions from images or videos. Salient Object Detection (SOD) can be divided into two subtasks: salient object localization and salient region segmentation. The former aims to locate the foreground subject in the image, while the latter is used to segment and refine the salient region based on the localization. Usually, a deep convolutional neural network is used to extract features, and then multiple layers of features are integrated to obtain the prediction results. However, simple feature processing cannot obtain enough structure and detailed information for accurate prediction. When only the true value is used as supervision, the extraction effect of the above information is not very ideal; some salient objects are missing or the background is mistakenly predicted as the foreground; the predicted salient results do not have fine edges, and even lose some edge information. Summary of the invention

[0006] One of the purposes of the present invention is to provide a salient target detection method and system integrating edge features, which is specifically used for salient target detection and has the ability to self-optimize intra-layer information. By effectively utilizing the complementarity between salient edge information and salient target information, the prediction accuracy of the salient target detection task is improved.

[0007] An embodiment of the present invention provides a salient object detection method integrating edge features, comprising:

[0008] The backbone network is used to process the image to obtain edge features and multiple groups of significant features;

[0009] The self-attention group pixel fusion module is used to process edge features and salient features to obtain salient edge features and salient target features.

[0010] The pooling model is used to process the salient features to obtain the global salient features;

[0011] A bidirectional feature fusion module is used to fuse salient edge features, salient target features and global salient features to obtain detection results.

[0012] Preferably, the backbone network includes: ResNet50; removing the fully connected layers of the last group of blocks of ResNet50, and marking the remaining convolution blocks as F1 to F5.

[0013] Preferably, the backbone network processes the image in the following steps:

[0014] The image is input into the backbone network, and a set of significant features of the output of each convolutional layer is obtained in sequence starting from the second convolutional layer, and the edge features of the output are obtained from the second convolutional layer.

[0015] Preferably, the self-attention group pixel fusion module processes edge features and saliency features in the following steps:

[0016] A self-attention group pixel fusion module is used to process edge features to obtain significant edge features;

[0017] Four self-attention group pixel fusion modules are used to process the four groups of salient features and obtain four groups of salient target features.

[0018] Preferably, the fusion steps of the bidirectional feature fusion module are as follows:

[0019] Down-sample the salient edge features and input the data into the first bidirectional feature fusion module together with the fourth set of salient target features and the global salient features;

[0020] The output of the first bidirectional feature fusion module, the third set of salient target features and the global salient features are input into the second bidirectional feature fusion module;

[0021] The output of the second bidirectional feature fusion module, the second set of salient target features, and the global salient features are input into the third bidirectional feature fusion module;

[0022] Input the output of the third bidirectional feature fusion module, the first set of salient target features and the global salient features into the fourth bidirectional feature fusion module;

[0023] The output of the fourth bidirectional feature fusion module is used as the detection result.

[0024] The present invention also provides a salient object detection system integrating edge features, comprising:

[0025] The first processing module is used to process the image using a backbone network to obtain edge features and multiple groups of significant features;

[0026] The second processing module is used to process the edge features and the salient features using the self-attention group pixel fusion module to obtain salient edge features and salient target features;

[0027] The third processing module is used to process the significant features using a pooling model to obtain global significant features;

[0028] The fusion detection module is used to fuse the significant edge features, significant target features and global significant features using a bidirectional feature fusion module to obtain the detection results.

[0029] Preferably, the backbone network includes: ResNet50; removing the fully connected layers of the last group of blocks of ResNet50, and marking the remaining convolution blocks as F1 to F5.

[0030] Preferably, the backbone network processes the image in the following steps:

[0031] The image is input into the backbone network, and a set of significant features of the output of each convolutional layer is obtained in sequence starting from the second convolutional layer, and the edge features of the output are obtained from the second convolutional layer.

[0032] Preferably, the self-attention group pixel fusion module processes edge features and saliency features in the following steps:

[0033] A self-attention group pixel fusion module is used to process edge features to obtain significant edge features;

[0034] Four self-attention group pixel fusion modules are used to process the four groups of salient features and obtain four groups of salient target features.

[0035] Preferably, the fusion steps of the bidirectional feature fusion module are as follows:

[0036] Down-sample the salient edge features and input the data into the first bidirectional feature fusion module together with the fourth set of salient target features and the global salient features;

[0037] The output of the first bidirectional feature fusion module, the third set of salient target features and the global salient features are input into the second bidirectional feature fusion module;

[0038] The output of the second bidirectional feature fusion module, the second set of salient target features, and the global salient features are input into the third bidirectional feature fusion module;

[0039] Input the output of the third bidirectional feature fusion module, the first set of salient target features and the global salient features into the fourth bidirectional feature fusion module;

[0040] The output of the fourth bidirectional feature fusion module is used as the detection result.

[0041] The present invention has the following beneficial effects:

[0042] First, the Self-Attention Group Pixel Fusion Module (SGPFM) is used for optimization within the network to enhance the representation of salient target features and edge features. This module generates feature groups with different receptive fields through the Self-Attention Group Pixel Fusion Module (SGPM), while the Self-Attention Group Fusion Module (SGFM) achieves the optimal combination.

[0043] Second, a bidirectional feature fusion (BFF) module is introduced to fully utilize the logical relationship between salient edge features and salient object features by cross-optimizing them. Through top-down optimization, this module enables the multi-scale features of salient objects to gradually acquire more details while suppressing background interference at salient edges.

[0044] 3. Bidirectional Self-Attention Edge Fusion Network (BSEFNet) outperforms the current state-of-the-art algorithms. This framework can not only accurately segment salient objects, but also accurately detect salient edges.

[0045] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0046] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0048] Figure 1 Schematic diagram of a salient object detection method integrating edge features in an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of a model architecture in an embodiment of the present invention;

[0050] Figure 3 Schematic diagram of a self-attention group pixel fusion module in an embodiment of the present invention;

[0051] Figure 4 Schematic diagram of a bidirectional feature fusion module in an embodiment of the present invention.

[0052] Figure 5 This is a performance comparison diagram in an embodiment of the present invention;

[0053] Figure 6 This is a diagram of visual comparison results in an embodiment of the present invention;

[0054] Figure 7 is a curve chart of comparison results in an embodiment of the present invention;

[0055] Figure 8 Schematic diagram of a salient object detection method integrating edge features in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0057] The embodiment of the present invention provides a salient object detection method integrating edge features, such as Figure 1 As shown, including:

[0058] The backbone network is used to process the image to obtain edge features and multiple groups of significant features;

[0059] The self-attention group pixel fusion module is used to process edge features and salient features to obtain salient edge features and salient target features.

[0060] The pooling model is used to process the salient features to obtain the global salient features;

[0061] A bidirectional feature fusion module is used to fuse salient edge features, salient target features and global salient features to obtain detection results.

[0062] The backbone network includes: ResNet50; remove the fully connected layers of the last group of blocks of ResNet50, and mark the remaining convolutional blocks as F1~F5.

[0063] Among them, the steps of the backbone network processing the image are as follows:

[0064] The image is input into the backbone network, and a set of significant features of the output of each convolutional layer is obtained in sequence starting from the second convolutional layer, and the edge features of the output are obtained from the second convolutional layer.

[0065] Among them, the processing steps of the self-attention group pixel fusion module for edge features and saliency features are as follows:

[0066] A self-attention group pixel fusion module is used to process edge features to obtain significant edge features;

[0067] Four self-attention group pixel fusion modules are used to process the four groups of salient features and obtain four groups of salient target features.

[0068] Among them, the fusion steps of the bidirectional feature fusion module are as follows:

[0069] Down-sample the salient edge features and input the data into the first bidirectional feature fusion module together with the fourth set of salient target features and the global salient features;

[0070] The output of the first bidirectional feature fusion module, the third set of salient target features and the global salient features are input into the second bidirectional feature fusion module;

[0071] The output of the second bidirectional feature fusion module, the second set of salient target features, and the global salient features are input into the third bidirectional feature fusion module;

[0072] Input the output of the third bidirectional feature fusion module, the first set of salient target features and the global salient features into the fourth bidirectional feature fusion module;

[0073] The output of the fourth bidirectional feature fusion module is used as the detection result.

[0074] The method of the present invention is implemented based on a network model. The model architecture follows the encoder-decoder structure and consists of four main components, namely the backbone Resnet50, DASPP, SGPFM and BFF. Figure 2 As shown in Figure 1, the overall framework consists of ResNet50, a dense spatial pyramid pooling model (DASPP), 5 self-attention group pixel fusion modules (SGPFM) and 4 bidirectional feature fusion (BFF) modules. The second layer of the backbone extracts edge information and then optimizes the two types of information interactively from top to bottom;

[0075] In the encoder part, a combination of backbone network, DASPP and 5 SGPFMs is adopted. The backbone network uses ResNet50, removes the fully connected layers of the last group of blocks, and marks the remaining convolutional blocks as F1~F5. Considering the high computational cost introduced by the fully connected layers, it is chosen to keep all the convolutional layers of ResNet50 as the backbone, because the convolutional blocks of F1~F5 can effectively extract multi-scale salient features. After the F2 block, an SGPFM is introduced to extract rough foreground edge features and supervised by salient edge labels. On the side of the F1~F5 blocks, 4 SGPFMs are constructed in order to extract multi-scale salient object features. In addition, on the top of the F5 block, a DASPP module (with dilation rates set to 1, 6, 12, 18) is applied to extract global salient features and supervised by segmentation labels. Next, these three features are transmitted to the decoder part for feature optimization.

[0076] The decoder part adopts a progressive optimization architecture to perform multi-scale feature aggregation and cross-optimization on edge features and object features. The decoder consists of four BFFs, which have similar structures. Taking BFF3 as an example, it has four inputs: salient object features and salient edge features output by BFF4, salient object feature side features output by SGPFM4, and global semantic saliency features sampled by DASPP. BFF3 fuses the salient object features of these four inputs and cross-optimizes them with the input salient edge features to produce refined features. It is worth noting that the features in F1 are discarded because they contain too much non-salient information. The final prediction map is output by BFF1 and supervised by salient labels and edge labels. The advantage of this structure is the multi-level and multi-scale optimization of salient features, which improves the performance of the model in the salient object detection task.

[0077] SGPFM aims to self-optimize the features in the convolutional layer and enhance the feature representation. Figure 3As shown in the figure, the SGPFM proposed in this paper contains two key parts: SGPM and SGFM. Self-attention group pixel module (SGPM) and self-attention group fusion module (SGFM); SGPM accepts input from backbone blocks F2-F5 and generates a set of features with gradually increasing receptive fields in a top-down feature fusion manner. SGFM achieves an effective combination of features from different receptive groupings. Each output of SGPFM1-SGPFM5 has the same number of channels. Compared with other feature refinement modules in the SOD task, SGPFM can output more representative features with the help of SGPM and SGFM.

[0078] For SGPM, its goal is to achieve self-optimization of features within the layer. First, the input of SGPFM is split into a set of features, denoted as {X j ,j=1,…,N}. Feature X1 and SGPFM i+1 The output of X is combined to integrate more semantic information. j Send to the self-attention convolutional layer for feature refinement. Because X j+1 (j=1, ..., N-1) and X j Combined with X j+1 The part undergoes more convolution operations, so the output group feature (denoted as F j ,j=1,…,N) will gradually increase its receptive field. Generally speaking, SGPM achieves semantic enhancement of intra-layer features.

[0079] The purpose of SGFM is to integrate the group features output by SGPM. As mentioned above, the group features have gradually increasing receptive fields. Group features with larger receptive fields have more global information, while features with smaller receptive fields contain more local details. Considering the complementarity of local and global information, we design a residual structure to integrate group features with different receptive fields. The general formula of SGFM is expressed as:

[0080] Y i =F i +Conv 3×3 (F i *F N+1-i ), i=1, ..., N# (1)

[0081] Where N is the number of splits in a group, + and * are the element-wise addition and multiplication of the average, respectively. Then, the grouping feature Y i Concatenate to produce the final output.

[0082] Bidirectional Feature Fusion (BFF) module, which is used to transmit messages between salient features and salient edge features in both directions;

[0083] like Figure 4 As shown in the figure, the three salient features are first aggregated through the fusion module and then cross-optimized with the edge features. By equipping these BFFs from top to bottom, the multi-scale edge and segmentation features can gradually obtain more details. First, the multi-scale salient features are aggregated using the feature fusion module. For each BFF i , the global semantic salient feature from DASPP is denoted as S g , from BFF i+1 The upper salient features of u , from SGPFM i The current layer salient features are denoted as S, and the output of the fusion module is denoted as S f Then the fusion process can be described as:

[0084]

[0085] Among them, F1, F2, and F3 are intermediate features, and Concat means that the features are cascaded in the channel dimension. High-level features contain rich semantic information, which can better locate the target and highlight the foreground object. Therefore, this paper proposes a feature fusion algorithm to optimize the salient features of the current layer by integrating multiple layers of salient features, which can not only reduce the background interference in the current layer features, but also enrich the semantic information. Then the fused salient features are cross-optimized with the edge features.

[0086] The cross-optimization part of BFF based on the logical relationship between SOD and SED is shown in the figure. i , from BFF i+1 The edge feature of is denoted as E, and the output edge feature and saliency feature are denoted as E * and S * Therefore, the process of cross optimization can be defined as:

[0087]

[0088] Where B1 is the middle edge feature, S f is the output of the fusion model. Operation E1*S f The shared parts of the salient features and edge features can be retained. Operation E * +S f -E * *S f The edges of significant features can be refined because operation E * +S f Makes up the outline of the significant features, while the operation-E * *S fThe extra edge parts introduced by the operation + can be removed. After configuring the BFF module from top to bottom, the salient target features can gradually obtain more refined edge information, and the background interference of the salient edge features can be effectively suppressed.

[0089] For efficient training, multi-supervision is used for both salient object detection and salient edge detection. Only a binary cross entropy (BCE) loss function is used to define the loss of the SED task. The BCE loss is written as:

[0090]

[0091] in Indicates prediction, Represents the true value.

[0092] Unlike BCE loss which is pixel-level supervision, CEL loss can consider global content and is proven to maintain the consistency of foreground highlights. Therefore, a mixture of BCE and CEL is used as the loss for the salient object detection task. CEL loss is expressed as:

[0093]

[0094] in In summary, the margin loss and saliency loss are denoted as:

[0095] L edge =L BCE #(6)

[0096]

[0097] The overall loss function is:

[0098]

[0099] in and are the saliency loss and edge loss of the middle layer respectively. The BCE loss and CEL loss in saliency supervision are balanced. The λ1 and λ2 defined in the formula balance the intermediate layer saliency loss and the intermediate layer edge loss respectively. They are empirically set to And λ1=λ2=0.6.

[0100] The model of the present invention is implemented on the public pytorch toolbox, and all experiments are performed using a single Nvidia-GTX3090 GPU. The pre-trained ResNet50 model is used to initialize the backbone network. The weights of other convolutional layers are initialized by a normal distribution with a standard deviation of 0.01 and a mean of zero. Some hyperparameters in the SGPFM module are set as follows: the number of splits N = 2 for SGPFM1 and SGPFM2, and the number of splits N = 4 for SGPFM3-SGPFM5. The batch size is set to 5, and the images are resized to 448*448 during the training and testing phases before being input into the network. The Adam optimizer is used to train our model for a total of 50 epochs. The learning rate is first set to 2e-5 and decayed by a factor of 0.5 at epoch14 and epoch22. To prevent gradient explosion, the gradients in the optimizer are clamped between [-0.5, 0.5].

[0101] For the saliency detection task, the popular DUTS-TR dataset is used to train our BSEFNet. DUTS-TE and three other popular datasets HKU-IS, ECSSD, and PACSALS are used to evaluate the performance of the model. DUTS is the largest salient object detection dataset, containing 10,553 images for training and 5,019 images for testing. Most images have large variations in position and scale. The PASCAL-S salient object dataset contains 850 images, all of which are selected from the object detection PASCAL VOC dataset. ECSSD contains 1,000 natural and meaningful semantic images with various complex scenes. These images are manually selected from the Internet. HKU-IS contains 4,447 images with high-quality annotations, many of which have multiple disconnected salient objects and low contrast.

[0102] To evaluate the quantitative performance, mean absolute error (MAE), F-measure, and precision-recall (PR) were used for comparison.

[0103] First, the prediction graph is thresholded into binary, and the precision and recall value pairs of different thresholds are calculated. The F-measure of different precision-recall pairs is calculated as follows:

[0104]

[0105] Following the main frequency tuning mechanism and the most mainstream algorithm contrast filtering algorithm, set β 2 = 0.3 to measure precision instead of recall. The maximum F-measure and average F-measure can be calculated as the maximum and average of the F-measures among all precision-recall pairs.

[0106] In addition, PR curves of different PR pairs are plotted to compare the algorithm of the present invention with other methods on four data sets. Formula (10):

[0107]

[0108] MAE is a common metric that can evaluate the difference between the predicted map and the true map. Denote the predicted saliency map and the corresponding ground truth as P and G respectively. The MAE score can be calculated as follows. Formula (11):

[0109]

[0110] Where W and H are the width and height of the image respectively.

[0111] In order to verify the effectiveness of the method, the BSEFNet of this application is compared with the most advanced methods. These methods include AFNet, BASNet, MINet, U 2 Net, ITSD, DSRNet, WSSA, ICON-R, M 3 Net-R, GFINet. Figure 5 As shown in Figure 2, it can be seen that the model performs well compared with advanced methods when comparing most of the evaluation indicators. Figure 6 Some visual comparison results with 4 other state-of-the-art models are shown. Figure 7 The PR curves and F-measure curves of some comparison methods on 4 data sets are plotted in the figure. The first two rows and the third and fourth rows are the PR curves and F-measure curves respectively; it can be seen that in the PR curves and F-measure curves, the solid red line representing the proposed method outperforms all other methods at most thresholds. By configuring the proposed SGPFMs module, the model of the present invention can perform self-optimization within the convolution layer, which will alleviate the problems of incorrect prediction of salient image targets and incomplete prediction of saliency. Due to the cross-optimization between salient target features and salient edge features, the model of the present invention configures the proposed

[0112] The BFFs module can generate sharp edges of foreground objects and suppress background interference in edge features.

[0113] In order to further improve the effect of salient object detection, in one embodiment, the salient object detection method integrating edge features further includes:

[0114] Extract features from the detection results and construct a preprocessing analysis set based on the extracted feature values; the extracted features include: features representing the shape of the salient area, features representing the pixels of each pixel in the salient area, etc.;

[0115] Matching the preprocessing analysis set with the call analysis set corresponding to each preprocessing template in the preconfigured preprocessing template call library;

[0116] Calling the preprocessing template calling library corresponding to the calling analysis set that matches the preprocessing analysis set;

[0117] The image input to the backbone network is preprocessed twice using the called preprocessing template, and the processed image is input to the backbone network again to obtain the secondary detection result.

[0118] By analyzing the primary detection results and selecting a suitable preprocessing template (mask template) to process the image and perform a secondary detection, the effect of salient target detection can be further improved.

[0119] Among them, the pre-processing template call library is constructed by professionals who analyze and construct a large amount of secondary detection data in advance. During the analysis and construction, after the image is input into the backbone network, it enters the primary processing. The results of the primary processing are processed by all the pre-processing templates and then output to the backbone network for secondary processing. Then professionals compare and analyze the results of the secondary processing to determine the best secondary processing result, and analyze and associate the mask template and the primary processing result corresponding to the secondary processing to achieve construction. In the constructed pre-processing template call library, the pre-processing template is associated with the call analysis set in a one-to-one correspondence. The call analysis set is extracted and constructed based on the detection results of the primary processing results in the constructed data.

[0120] In one embodiment, a salient object detection method integrating edge features includes:

[0121] Extract features of environmental data monitored in the environment of image acquisition to obtain a first preprocessing parameter set;

[0122] The calling parameter set is matched with the calling sets of each preprocessing method in the preset preprocessing method determination library to determine the preprocessing method.

[0123] Among them, the calling parameter set is formed by arranging the features extracted from the environmental data in a preset order, and the features extracted from the environmental data include: characteristic values ​​representing temperature, characteristic values ​​representing humidity, characteristic values ​​representing light intensity, characteristic values ​​representing the distribution of particles in the air, etc.; the preprocessing method determination library is constructed in advance, and the image preprocessing method is correspondingly associated with the calling set in the library. The calling set and the calling parameter set have the same or similar composition. The two are matched to realize the calling of the preprocessing method in the preprocessing method determination library.

[0124] This embodiment analyzes the data of monitoring the environment in which the image is taken, and calls an accurate image processing method to eliminate the influence of the environment on the image shooting as much as possible, so as to ensure the accuracy of significant target detection. Environmental data include: light intensity, temperature, humidity, distribution of particles in the air, etc. Preprocessing methods include: brightness adjustment value, etc. Furthermore, the spatial layout of the surrounding space can be analyzed, and specifically, the objects arranged in the space can be quantitatively analyzed to obtain a characteristic value representing the distribution of objects in the space, and the characteristic value includes a coding part representing the type of object and a coding part representing the direction and distance of the object from the shooting position.

[0125] The present invention also provides a salient object detection system integrating edge features, such as Figure 8 As shown, including:

[0126] The first processing module 1 is used to process the image using a backbone network to obtain edge features and multiple groups of significant features;

[0127] The second processing module 2 is used to process the edge features and the salient features using the self-attention group pixel fusion module to obtain salient edge features and salient target features;

[0128] The third processing module 3 is used to process the significant features using a pooling model to obtain global significant features;

[0129] The fusion detection module 4 is used to fuse the significant edge features, the significant target features and the global significant features using a bidirectional feature fusion module to obtain the detection result.

[0130] The backbone network includes: ResNet50; remove the fully connected layers of the last group of blocks of ResNet50, and mark the remaining convolutional blocks as F1~F5.

[0131] Among them, the steps of the backbone network processing the image are as follows:

[0132] The image is input into the backbone network, and a set of significant features of the output of each convolutional layer is obtained in sequence starting from the second convolutional layer, and the edge features of the output are obtained from the second convolutional layer.

[0133] Among them, the processing steps of the self-attention group pixel fusion module for edge features and saliency features are as follows:

[0134] A self-attention group pixel fusion module is used to process edge features to obtain significant edge features;

[0135] Four self-attention group pixel fusion modules are used to process the four groups of salient features and obtain four groups of salient target features.

[0136] Among them, the fusion steps of the bidirectional feature fusion module are as follows:

[0137] Down-sample the salient edge features and input the data into the first bidirectional feature fusion module together with the fourth set of salient target features and the global salient features;

[0138] The output of the first bidirectional feature fusion module, the third set of salient target features and the global salient features are input into the second bidirectional feature fusion module;

[0139] The output of the second bidirectional feature fusion module, the second set of salient target features, and the global salient features are input into the third bidirectional feature fusion module;

[0140] Input the output of the third bidirectional feature fusion module, the first set of salient target features and the global salient features into the fourth bidirectional feature fusion module;

[0141] The output of the fourth bidirectional feature fusion module is used as the detection result.

[0142] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A salient object detection method integrating edge features, characterized in that: include: The backbone network is used to process the image to obtain edge features and multiple groups of significant features; The self-attention group pixel fusion module is used to process edge features and salient features to obtain salient edge features and salient target features. The pooling model is used to process the salient features to obtain the global salient features; A bidirectional feature fusion module is used to fuse salient edge features, salient target features and global salient features to obtain detection results. The backbone network includes: ResNet50; remove the fully connected layers of the last group of blocks of ResNet50, and mark the remaining convolutional blocks as F1~F5; The steps of the backbone network to process the image are as follows: Input the image into the backbone network, and obtain a set of significant features of the output of each convolutional layer starting from the second convolutional layer, and obtain the edge features of the output from the second convolutional layer; The processing steps of the self-attention group pixel fusion module for edge features and saliency features are as follows: A self-attention group pixel fusion module is used to process edge features to obtain significant edge features; Four groups of salient features are processed using four self-attention group pixel fusion modules to obtain four groups of salient target features; The fusion steps of the bidirectional feature fusion module are as follows: Down-sample the salient edge features and input the data into the first bidirectional feature fusion module together with the fourth set of salient target features and the global salient features; Input the output of the first bidirectional feature fusion module, the third set of salient target features and the global salient features into the second bidirectional feature fusion module; Input the output of the second bidirectional feature fusion module, the second set of salient target features and the global salient features into the third bidirectional feature fusion module; Input the output of the third bidirectional feature fusion module, the first set of salient target features and the global salient features into the fourth bidirectional feature fusion module; The output of the fourth bidirectional feature fusion module is used as the detection result; Among them, a mixture of BCE and CEL is used as the loss of the salient object detection task; The BCE loss is written as: in ∈[0,1] H×W×1 Indicates prediction, ∈{0,1} H×W×1 represents the true value; and are the total number of pixels of the width and height of the image respectively; The CEL loss is expressed as: in ∈ , ∈ ; In summary, the edge loss and significance loss are recorded as: The overall loss function is: in and They are the edge loss and saliency loss of the middle layer respectively; =0.8,λ1=λ2=0.

6.

2. A salient object detection system integrating edge features, characterized in that: include: The first processing module is used to process the image using a backbone network to obtain edge features and multiple groups of significant features; The second processing module is used to process the edge features and the salient features using the self-attention group pixel fusion module to obtain salient edge features and salient target features; The third processing module is used to process the significant features using a pooling model to obtain global significant features; A fusion detection module is used to fuse significant edge features, significant target features and global significant features using a bidirectional feature fusion module to obtain a detection result; The backbone network includes: ResNet50; remove the fully connected layers of the last group of blocks of ResNet50, and mark the remaining convolutional blocks as F1~F5; The steps of the backbone network to process the image are as follows: Input the image into the backbone network, and obtain a set of significant features of the output of each convolutional layer starting from the second convolutional layer, and obtain the edge features of the output from the second convolutional layer; The processing steps of the self-attention group pixel fusion module for edge features and saliency features are as follows: A self-attention group pixel fusion module is used to process edge features to obtain significant edge features; Four groups of salient features are processed using four self-attention group pixel fusion modules to obtain four groups of salient target features; The fusion steps of the bidirectional feature fusion module are as follows: Down-sample the salient edge features and input the data into the first bidirectional feature fusion module together with the fourth set of salient target features and the global salient features; Input the output of the first bidirectional feature fusion module, the third set of salient target features and the global salient features into the second bidirectional feature fusion module; Input the output of the second bidirectional feature fusion module, the second set of salient target features and the global salient features into the third bidirectional feature fusion module; Input the output of the third bidirectional feature fusion module, the first set of salient target features and the global salient features into the fourth bidirectional feature fusion module; The output of the fourth bidirectional feature fusion module is used as the detection result; Among them, a mixture of BCE and CEL is used as the loss of the salient object detection task; The BCE loss is written as: in ∈[0,1] H×W×1 Indicates prediction, ∈{0,1} H×W×1 represents the true value; and are the total number of pixels of the width and height of the image respectively; The CEL loss is expressed as: in ∈ , ∈ ; In summary, the edge loss and significance loss are recorded as: The overall loss function is: in and They are the edge loss and saliency loss of the middle layer respectively; =0.8,λ1=λ2=0.6.

Images