Salient object detection method and system based on type clue correlation guidance
By combining the significance object detection method of convolutional neural network and capsule network, the problem of unconsidered relationship between regions is solved, higher detection accuracy and completeness are achieved, and network complexity is reduced.
Patent Information
- Application Number
- CN202310958480.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-31
- Publication Date
- 2025-08-26
- Estimated Expiration
- 2043-07-31
AI Technical Summary
The existing deep learning-based significance object detection method fails to effectively consider the relationship between regions when independently calculating significant areas in the image, resulting in insufficient object integrity.
Combining convolutional neural networks and capsule networks, by integrating the clues of contrast and component-total relationships, HRNet extracts multi-scale features, performs significant predictions, and trains the network through type interaction modules and error functions to enhance the accuracy of detection.
Improves the accuracy and completeness of significance object detection, reduces the amount of network parameters and speeds up the inference, showing higher F-measure and lower mean errors.
Smart Images
Figure CN117011602B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a salient object detection method and system based on type cue correlation guidance. Background Art
[0002] The task of salient object detection is to mimic human visual perception and automatically identify and segment attractive regions or objects. It helps capture informative regions that contain key scene semantics, saving significant computational complexity for other tasks, including segmentation, image fusion, image retrieval, and object recognition. Early salient object detection methods primarily extracted handcrafted features to identify contrasting regions. The development of deep learning has significantly broken through the bottlenecks of handcrafted methods and will continue to bring about remarkable progress.
[0003] Salient object detection methods based on deep learning primarily rely on deep neural networks, particularly convolutional neural networks, to extract discriminative features and identify salient regions with high contrast around them. These methods excel at capturing object details. However, these methods independently compute salient regions within an image without considering the interrelationships between regions, which compromises the integrity of the objects. To address this issue, researchers have recently attempted to incorporate part-whole relationships into salient object detection. For example, Q. Zhang, M. Duanmu, Y. Luo, Y. Liu, and J. Han, “Engaging part-whole hierarchies and contrast cues for salient object detection,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 6, pp. 3644–3658, 2021, propose a mutual attention mechanism that integrates contrast and part relationships to more accurately detect salient objects. Despite initial success, this approach remains in its infancy and several challenges remain to be addressed. For example, due to a lack of guidance from integrating contrast cues and part-whole relationships, the results cannot guarantee the integrity of salient objects. Summary of the Invention
[0004] To address the shortcomings of existing methods, the present invention provides a salient object detection method based on the complementary properties of contrast and part-whole relationships caused by deep neural networks such as convolutional neural networks and capsule networks. The method integrates the correlation of these two cues from convolutional neural networks and capsule networks and enables them to interact with each other, thereby increasing the detection accuracy of the salient object detection method based on the complementary properties of contrast and part-whole relationships.
[0005] The technical solution adopted by the present invention is: a salient object detection method based on type clue correlation guidance includes the following steps:
[0006] Step 1: Use the HRNet backbone to extract deep features of the image to be detected;
[0007] Furthermore, step one specifically includes:
[0008] The three-channel RGB image is input into the HRNet backbone to extract multi-scale and high-resolution features, and the features are input into the dilated spatial pyramid pooling module with different expansion rates to obtain a feature map containing contextual information.
[0009] Step 2: Input the context information feature map into the convolutional neural network to obtain the contrast saliency prediction result;
[0010] Furthermore, step 2 specifically includes:
[0011] The convolutional neural network branch includes stages 1 to 4. The convolution in stages 1 and 2 uses a 3×3 convolution kernel with a stride of 2, and the convolution in stage 3 uses a 1×1 convolution kernel with a stride of 1. In stage 4, batch normalization is used to normalize the data, and the ReLU activation function is used to prevent the gradient from disappearing. Finally, the contrast saliency prediction result F is obtained through a 1×1 convolution operation. CS .
[0012] Step 3: Input the context information feature map into the capsule network to obtain the part-whole relationship significance prediction result;
[0013] Furthermore, step three specifically includes:
[0014] The capsule network consists of a main capsule layer, a convolutional capsule layer, and a capsule-like layer. Each layer contains 8 types of capsules. The activation output of the capsule-like layer is used as the capsule feature, and the part-whole relationship significance prediction result F is obtained through convolution calculation. PO .
[0015] Step 4: Contrast saliency prediction result F CS and the prediction results of the significance of the part-whole relationship F PO Calculation of horizontal and vertical voting matrices;
[0016] Furthermore, step four specifically includes:
[0017] Step 41: Calculate F CS and F PO Spatial correlation in the horizontal and vertical dimensions;
[0018] The formula for spatial correlation in the horizontal dimension is:
[0019]
[0020] Spatial correlation formula in the vertical dimension:
[0021]
[0022] in, is the horizontal feature operation result matrix, F CS is the contrast saliency prediction result, F PO The prediction results of the significance of the part-whole relationship;
[0023] Step 42: Calculate the prior consistency between the contrast saliency prediction result and the component-whole relationship saliency prediction result. The formula is:
[0024]
[0025] Among them, F CS1 and F PO1 They are the prior consistency of the contrast saliency prediction results and the component-whole relationship saliency prediction results;
[0026] Step 43: Calculate the significance prediction of the contrast significance prediction result and the component-whole relationship significance prediction result. The formula is:
[0027] F C ' S =F CS +F CS ⊙F CS1 (4)
[0028] F' PO =F PO +F PO ⊙F PO1
[0029] Among them, ⊙ is the exclusive OR operation, F' CS and F' PO They are saliency prediction related to contrast cue type and saliency prediction related to part-whole relationship, respectively.
[0030] Step 5: Predict the saliency of contrast cue type F' CS Significant predictions related to part-whole relationships F' PO to integrate;
[0031] Furthermore, step five specifically includes:
[0032] Step 51: Use the type interaction module to predict the saliency of the contrast cue type F' CS Significant predictions related to part-whole relationships F'PO The fusion is performed to obtain the output graph M1 of the type interaction module; wherein, the fusion formula is:
[0033]
[0034] Where GAP(·) represents global average pooling, Conv31D(·) represents one-dimensional convolution with a kernel size of 3×3, σ represents s-type operation; F' CS and F' PO They are the saliency prediction related to contrast cue type and the saliency prediction related to part-whole relationship, respectively. M1 represents the output map of the type interaction module;
[0035] Step 52: Obtain the output graphs M2 and M3 of the type interaction module through the same fusion method. After fusing and convolving M1 and M2, perform fusion and convolution with M3 to obtain the final saliency graph M. O .
[0036] Step 6: Use the cross entropy error function and the boundary error function to jointly train the dual-branch target detection network;
[0037] Furthermore, the formula for the combination of the cross entropy error function and the boundary error function is:
[0038]
[0039] in, represents the weighted cross entropy loss function, represents the boundary error function.
[0040] Furthermore, the formulas for the weighted cross entropy loss function and the boundary error function are:
[0041]
[0042]
[0043] Among them, γ is a hyperparameter, l is a calibration function, is the pixel difference correction parameter at horizontal position i and vertical position j, and denotes the position of each pixel on the predicted value and the ground truth value respectively, Ψ denotes all the parameters of the model, is the predicted probability.
[0044] Furthermore, the salient object detection system based on type clue correlation guidance includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement a salient object detection method based on type clue correlation guidance.
[0045] Furthermore, a computer-readable medium storing computer program code implements a salient object detection method guided by type cue relevance when the computer program code is executed by a processor.
[0046] Beneficial effects of the present invention:
[0047] 1. It fully utilizes the contrast and part-whole relationship caused by deep neural networks such as convolutional neural networks and capsule networks, two complementary semantic properties for deep salient object detection, to produce better saliency cues;
[0048] 2 The proposed method is compared with the existing methods on five data sets. The F-measure value and MAE value of the proposed method are better than those of the existing methods, and the network parameters are smaller and the inference speed is faster. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 This is a logic block diagram of the salient object detection method based on type clue correlation guidance of the present invention;
[0050] Figure 2 It is a simulation diagram of the experimental results of the present invention and the prior art under the ECSSD database. DETAILED DESCRIPTION
[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore only shows the components related to the present invention.
[0052] like Figure 1 As shown, the salient object detection method and system based on type clue correlation guidance includes the following steps:
[0053] Step 1: Use the HRNet backbone to extract deep features of the image to be detected;
[0054] Given a three-channel RGB image, it is fed into the HRNet backbone to extract multi-scale and high-resolution features, and the features are further fed into the dilated spatial pyramid pooling module with different dilation rates (1, 6, 12, and 18) to capture rich contextual information; then, all feature maps are fused to obtain a feature map containing contextual information (88×88×128).
[0055] Step 2: Input the context information feature map into the convolutional neural network to obtain the contrast saliency prediction result;
[0056] The convolutional neural network branch includes stages 1 to 4. The convolution in stages 1 and 2 uses a 3×3 convolution kernel with a stride of 2, while the convolution in stage 3 uses a 1×1 convolution kernel with a stride of 1. In stage 4, batch normalization is used to normalize the data, and the ReLU activation function is used to prevent the gradient from disappearing. Finally, the contrast saliency prediction result F is obtained through a 1×1 convolution operation. CS (22×22×1);
[0057] The input and output of each stage are expressed as:
[0058] F out =ReLU(BN(Conv3(F in )) (1)
[0059] Among them, F in and F out Represents the input and output of the convolution stage.
[0060] Step 3: Input the context information feature map into the capsule network to obtain the part-whole relationship significance prediction result;
[0061] The capsule network branch aims to enhance the object integrity of the feature map contained in the backbone network; the capsule network consists of: a main capsule layer, a convolutional capsule layer and a capsule-like layer. Each layer contains 8 types of capsules. The activation output of the capsule-like layer is used as the capsule feature (22×22×8×1). Through convolution, further calculation is performed to obtain the part-whole relationship significance prediction result F PO (22×22×1).
[0062] Step 4: Contrast saliency prediction result F CS and the prediction results of the significance of the part-whole relationship F PO Calculation of horizontal and vertical voting matrices;
[0063] The spatial correlation formula in the horizontal dimension is:
[0064]
[0065] Among them, SC H Refers to the spatial correlation of two saliency priors in the horizontal dimension, is the horizontal feature operation result matrix.
[0066] Spatial correlation formula in the vertical dimension:
[0067]
[0068] Among them, SC V refers to the spatial correlation between the two saliency priors in the vertical dimension, is the horizontal feature operation result matrix.
[0069] Correlation Guidance: Spatial Correlation SC H and SC V Can be used to guide the significance prior F CS and F PO , to improve their significant properties, the formula is as follows:
[0070]
[0071] Among them, F CS1 and F PO1 They are the prior consistency of the contrast saliency prediction results and the part-whole relationship saliency prediction results, respectively.
[0072] Self-attention: To improve the two saliency predictions, namely F CS and F PO The salience properties of , which are determined by the relevant perceptual salience prior self-participation, can be expressed as:
[0073] F' CS =F CS +F CS ⊙F CS1 (5)
[0074] F' PO =F PO +F PO ⊙F PO1
[0075] Among them, ⊙ is the exclusive OR operation, F' CS and F' PO By involving F CS and F PO The spatial correlation between the two saliency predictions is used to improve the saliency attribute; the saliency prediction F' based on type correlation is obtained using formulas (2)-(5) CS and F' PO .
[0076] Step 5: Obtain the saliency prediction F' related to contrast cues and part-whole type CS and F' PO , so that the two types of information interact and integrate, an attention mechanism type interaction module is established, which can be expressed as:
[0077]
[0078] Among them, GAP(·) represents global average pooling, Conv31D(·) represents a one-dimensional convolution with a kernel size of 3×3, σ represents an s-type operation; α' is a CSThe result of preprocessing of saliency prediction, and then interaction after preprocessing, F' CS and F' PO They are the saliency prediction related to the contrast cue type and the saliency prediction related to the part-whole relationship, respectively. M1 represents the feature prediction result of the type interaction module.
[0079] After the type interaction module is completed, three feature predictions M1, M2 and M3 are obtained, and convolution is fused step by step to obtain the final saliency map M O .
[0080] Among them, M2 and M3 are F' CS and F' PO After upsampling, the feature prediction results obtained by applying the type interaction module formula (6) are obtained.
[0081] Step 6: Use the cross entropy error function and the boundary error function to jointly train the dual-branch target detection network. The error function l oss for:
[0082]
[0083] in, represents the weighted cross entropy loss function, Represents the boundary error function; the specific formula is as follows:
[0084]
[0085]
[0086] Among them, γ is a hyperparameter; is the pixel difference correction parameter of horizontal position i and vertical position j, if If is larger, the pixel at (i, j) is obviously different from the surrounding environment, so it is an important pixel that deserves more attention. On the contrary, if Very small, considered as an ordinary pixel, l is the calibration function, and denotes the position of each pixel on the predicted value and the ground truth value respectively, Ψ denotes all the parameters of the model, is the predicted probability.
[0087] The following is a further explanation of the technical effects of the present invention in conjunction with simulation experiments:
[0088] Simulation conditions:
[0089] The experiment was implemented in pytorch. The base network was initialized with HRNet pre-trained weights, and other weights were initialized with truncated normal random (σ = 0.01). The bias was initialized to 0. The SGD optimizer was used to train the model, and the initial learning rate was 10. -3 ; The DUTS training dataset is selected to train the network, and horizontal flipping is used to enhance the data.
[0090] Simulation content and result analysis:
[0091] Simulation 1:
[0092] The present invention is compared with the existing salient object detection method based on the fusion of contrast cues and part-whole relationship cues (Q. Zhang, M. Duanmu, Y. Luo, Y. Liu, and J. Han, “Engaging part-whole hierarchies and contrast cues for salient object detection,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 6, pp. 3644-3658, 2021). The salient object detection experiment is conducted on the public image dataset ECSSD, and some results are intuitively compared; Figure 2 As shown in the figure, the original image represents the image in the database used for experimental input, and the true value image represents the manually calibrated binary image; Figure 2 It can be seen that compared with the existing technology, the present invention has better integrity in salient target detection, better background suppression effect, and better effect in salient target detection in complex scenes.
[0093] Simulation 2:
[0094] The results of the salient object detection experiment of the present invention and the existing salient object detection methods based on contrast cues and part-whole relationship cues were obtained on the public image database ECSSD. The results were objectively evaluated using recognized evaluation indicators, namely the weighted average F-measure value and MAE value. The evaluation simulation results are shown in Table 1.
[0095] Table 1 Comparison between the present invention and prior art
[0096]
[0097] As can be seen from Table 1, compared with the existing technology, the present invention has higher F-measure, S-measure, E-measure values and lower MAE value, which shows that the present invention has better completeness and consistency in salient object detection, fully demonstrating the effectiveness and superiority of the method of the present invention.
[0098] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.
Claims
1. A salient object detection method based on type clue correlation guidance, characterized by: The following steps are involved: Step 1: Use the HRNet backbone to extract deep features of the image to be detected; Step 1 specifically includes: The three-channel RGB image is input into the HRNet backbone to extract multi-scale and high-resolution features. The features are then input into the dilated spatial pyramid pooling module with different expansion rates to obtain a feature map containing contextual information. Step 2: Input the context information feature map into the convolutional neural network to obtain the contrast saliency prediction result; Step 3: Input the context information feature map into the capsule network to obtain the part-whole relationship significance prediction result; Step 4: Calculate the horizontal and vertical voting matrices of the contrast significance prediction results and the component-whole relationship significance prediction results; Step 5: Fusion of saliency predictions related to contrast cue type and part-whole relationship; Step 4 specifically includes: Step 41: Calculation F CS and F PO Spatial correlation in the horizontal and vertical dimensions; The formula for spatial correlation in the horizontal dimension is: （1） Spatial correlation formula in the vertical dimension: （2） in, is the horizontal feature operation result matrix, F CS is the contrast saliency prediction result, F PO The prediction results of the significance of the part-whole relationship; Step 42: Calculate the prior consistency between the contrast saliency prediction result and the component-whole relationship saliency prediction result. The formula is: （3） in, and They are the prior consistency of the contrast saliency prediction results and the component-whole relationship saliency prediction results; Step 43: Calculate the significance prediction of the contrast significance prediction result and the component-whole relationship significance prediction result. The formula is: ⊙ （4） ⊙ Among them, ⊙ is the XOR operation, and They are respectively saliency prediction related to contrast cue type and saliency prediction related to part-whole relationship; Step 6: Use the cross entropy error function and the boundary error function to jointly train the dual-branch target detection network; Step 5 specifically includes: Step 51: Use the type interaction module to predict the saliency of contrast cue types Significant predictions related to part-whole relationships Perform fusion to obtain the output graph of the type interaction module ; Among them, the fusion formula is: （5） in, represents global average pooling, represents a one-dimensional convolution with a kernel size of 3×3, and σ represents an s-type operation; and They are respectively the saliency prediction related to contrast cue type and the saliency prediction related to part-whole relationship, Output graph of the representation type interaction module; Step 52: Obtain the output graph of the type interaction module through the same fusion method and ,Will and After fusion convolution and Perform fusion convolution to obtain the final saliency map .
2. The salient object detection method based on type clue correlation guidance according to claim 1, characterized in that: Step 2 specifically includes: The convolutional neural network branch includes stages 1 to 4. The convolution in stages 1 and 2 uses a 3×3 convolution kernel with a stride of 2, and the convolution in stage 3 uses a 1×1 convolution kernel with a stride of 1. In stage 4, batch normalization is used to normalize the data, and the ReLU activation function is used to prevent the gradient from disappearing. Finally, the contrast saliency prediction result is obtained through a 1×1 convolution operation. F CS .
3. The salient object detection method based on type clue correlation guidance according to claim 1, characterized in that: The capsule network branch aims to enhance the object integrity of the feature map contained in the backbone network; the capsule network consists of: a main capsule layer, a convolutional capsule layer and a capsule-like layer. Each layer contains 8 types of capsules. The activation output of the capsule-like layer is used as the capsule feature, which is further calculated through convolution to obtain the part-whole relationship significance prediction result. F PO .
4. The salient object detection method based on type clue correlation guidance according to claim 1, characterized in that: The formula for the combination of the cross entropy error function and the boundary error function is: （6） in, represents the weighted cross entropy loss function, represents the boundary error function.
5. The salient object detection method based on type clue correlation guidance according to claim 4, characterized in that: The formulas for the weighted cross entropy loss function and the boundary error function are: （7）； （8） Among them, γ is a hyperparameter, is the calibration function, Horizontal position i vertical position j Pixel difference correction parameters, is the calibration function, and Represents the position of each pixel on the predicted value and the ground truth value, respectively, represents all parameters of the model, is the predicted probability.
6. Salient object detection system based on type clue correlation guidance, characterized by: include: a memory for storing instructions executable by the processor; A processor, configured to execute instructions to implement the salient object detection method based on type clue correlation guidance according to any one of claims 1 to 5.
7. A computer-readable medium storing computer program code, characterized in that When the computer program code is executed by a processor, the computer program code implements the salient object detection method based on type clue correlation guidance according to any one of claims 1 to 5.