Significance target detection method based on mixed feature superposition cross attention

By introducing a cross attention mechanism and a two-way information refinement module in significance target detection, combined with a U-shaped network, the problems of insufficient recognition capabilities and unused edge information complementarity in low-contrast scenarios are solved, and more accurate and complete Significant target segmentation.

CN119068182APending Publication Date: 2024-12-03GUANGDONG UNIVERSITY OF FOREIGN STUDIES
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310632184.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing significance object detection methods lack recognition capabilities in low-contrast scenarios, and the complementarity of edge information and significant regions are not fully utilized. Convolutional operations lead to loss of feature information, affecting the segmentation effect.

Method used

Using the object detection method based on mixed features superimposed cross attention, multi-level features are extracted through residual neural networks, cross attention mechanism is introduced to obtain context information, bidirectional information refinement modules integrate edges and significant features, and optimized results are output using U-shaped networks. , combined with binary cross entropy loss for model correction.

Benefits of technology

It improves segmentation accuracy in low-contrast scenarios, optimizes edge details and segmentation integrity, generates higher quality significance and edge feature maps, and improves the overall effect of significance target detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068182A_ABST
    Figure CN119068182A_ABST
Patent Text Reader

Abstract

The invention discloses a mixed feature superposition-based cross attention refinement target detection method, which comprises the following steps of: A, inputting an RGB image and a mask mark GT image of an original image, and automatically cutting the resolution of the input image; b, inputting a residual neural network (Resnet) to obtain basic features of different levels of the RGB image; c, adopting a position attention mechanism to obtain a feature map fused with attention weights; d, executing a multi-task strategy, and performing independent feature extraction on the feature map by using two convolutional layers for salient target detection and edge detection; e, executing a bidirectional refinement strategy, and carrying out iteration refinement on feature maps output by different levels of the two detection tasks in the step D for multiple times; f, executing a U-shaped network segmentation strategy according to a feature map obtained by up-sampling and cascading different level features to obtain a predicted saliency map and an edge map; g, adopting binary cross entropy loss, calculating a loss function by using the true value graph and the prediction graph, and performing back propagation to correct model parameters; and H, repeatedly executing the steps B to G, correcting the model until a set round is finished, and outputting a final model. According to the method, the saliency target segmentation performance in a low-contrast scene can be remarkably improved, and the problems of partial missing and error detection of the saliency target are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a salient object detection method, which can be applied to the fields of video object segmentation, autonomous driving, and semantic segmentation. Background Art

[0002] The real world is full of information. When people use vision to perceive the outside world, they are usually selective, focusing on more important targets and ignoring a lot of irrelevant information. Such targets are called salient targets. The main work of salient target detection is to simulate the human visual system by establishing a visual attention model, identify the salient areas that people notice at first glance in a natural image, identify useful information and segment the salient areas. Traditional salient target detection methods generally use bottom-level semantic information such as contour, color, texture and shape features to perform data-driven detection from top to bottom. Such methods have good recognition capabilities for natural images containing single targets or high contrast, but traditional methods are usually not applicable to low contrast, complex background, small target scenes and large target scenes.

[0003] In recent years, research on salient object detection has achieved certain results. The relevant methods are all based on deep learning and can be further divided into optimization methods based on additional data assistance, weak supervisor learning methods, and optimized context information extraction methods. The optimization methods based on additional data assistance are mainly based on edge detection assistance. Wu et al. proposed to use three tasks of salient object detection, foreground contour detection, and edge detection to design the network at the same time. Subsequently, in 2019, Liu et al. proposed PoolNet combined with edge detection to capture edge information to enhance the details of salient objects. Luo et al. proposed a U-Net-based architecture, which contains IOU edge loss and uses edge cues to detect salient objects. This type of method enriches the edge area of ​​the salient map by introducing edge information, but does not consider the complementarity between salient areas and edge details. Weak supervisor learning methods mainly consider the high cost of pixel-level annotated data. Zeng et al. designed a weakly supervised source learning architecture that integrates multiple weakly supervised cues to detect salient objects based on image-level labels, image descriptions, and unlabeled data. This type of method is suitable for situations where data samples are insufficient, but when the amount of data is sufficient, the features extracted by the weakly supervised learning method are not as rich as those extracted by the deep convolutional neural network, so the recognition effect is not good. Optimized context information extraction methods focus on the combination of local and global information. Liu proposed a context attention network PiCANet to learn context features, embedding it into a U-Net network, integrating global context and multi-scale local context to enhance position and direction information. However, this method lacks attention to edges or small parts, resulting in poor segmentation results.

[0004] In summary, the existing salient object detection methods have limitations in practical use, so it is necessary to improve them. Summary of the invention

[0005] The present invention proposes an object detection method based on hybrid feature superposition and cross attention. The method introduces a cross attention module (CCA) to effectively obtain useful contextual information of surrounding pixels from long-distance dependencies. The model fully considers the complementarity between saliency detection and edge information, realizes information fusion through a bidirectional information refinement module (BIRM), and refines the multi-level features of saliency object detection and edge detection at the same time. Finally, a U-shaped network is used to output the optimization result.

[0006] The technical problem solved by the present invention can be achieved by adopting the following technical solutions: The present invention provides a target detection method based on mixed feature superposition and cross attention, comprising the following steps: A. Input the RGB image of the original image, the masked GT image, and automatically crop the resolution of the input image; B. Calculate different levels of features of RGB images through residual neural network (Resnet); C. Use the position attention mechanism to obtain the feature map of fused attention weights; Specifically, by calculating the vertical and horizontal feature weights of feature maps at different levels, three sets of different feature map outputs are obtained at each level, and then the feature maps are spliced ​​on the channels, and finally connected to a convolutional layer to obtain feature maps with feature weights.

[0007] D. Implement a multi-task strategy and use two convolutional layers to extract independent features from feature maps for salient object detection and edge detection. E. Execute the bidirectional refinement strategy, and iterate and refine the feature maps output at different levels of the two detection tasks in step D multiple times; Specifically, feature maps are generated at four different levels, and the feature maps at the lower level are superimposed with the feature maps at the upper level, that is, the feature maps at the upper level are downsampled, and then a convolutional layer is used to output a feature map with the same number of channels as the original feature map at that level.

[0008] F. Based on the feature map obtained by upsampling and concatenating features at different levels, a U-shaped network segmentation strategy is implemented to obtain the predicted saliency map and edge map; G. Use binary cross entropy loss, use the true value graph and the predicted graph to calculate the loss function, and perform back propagation to correct the model parameters; Specifically, the loss function includes a saliency map loss function, an edge detection loss function, and a saliency map and edge detection weighted loss function, and three different loss functions are optimized and calculated.

[0009] H. Repeat steps B to G to revise the model until the set rounds are completed and output the final model.

[0010] The present invention aims to propose a salient target detection method based on hybrid feature superposition and cross-attention, construct a residual neural network to extract multi-level feature maps rich in semantic information, adopt a cross-attention mechanism to obtain a feature map of fused feature weights; utilize the mutual logical relationship and complementarity between edge features and salient map segmentation features, consider the influence between different feature levels, and realize the cross-fusion of different features by cascading and superimposing four bidirectional information refinement modules; utilize the salient segmentation map to limit and generate higher quality edge features, and at the same time fuse high-quality edge features and salient feature maps to generate more accurate and complete segmentation maps, implement a back-propagation strategy, correct model parameters according to the loss function, and obtain the final output model through multiple training iterations. Its characteristics and advantages are: Aiming at the poor performance of existing salient object detection methods in recognizing salient objects, a salient object detection method based on hybrid feature superposition and cross attention was proposed.

[0011] Optimize the problem of edge detail information loss. The mainstream salient object detection model can segment the main module of the salient area, but the details near the edge are not processed very well. For this reason, this paper introduces a two-way information refinement module, which aims to utilize the complementarity of edge information and salient areas to improve the problem of edge detail information loss.

[0012] The optimization prediction target is partially missing, which is the segmentation integrity problem. Most existing segmentation models use convolutional neural networks for feature extraction, and some information is lost in the convolution process. To improve this problem, this paper introduces the idea of ​​residual learning in the bidirectional information refinement module to reduce the information lost due to convolution.

[0013] Optimize the model's insufficient recognition ability in low-contrast scenes. In low-contrast scenes, the model usually recognizes the foreground and part of the background as the same object. This is because the existing model has a weak perception of global information. This paper introduces a cross-attention mechanism, which uses the attention model to obtain dense contextual information from all pixels, fuse the attention weights, and output a more discriminative feature map.

[0014] The salient object detection method provided by the present invention can be used to segment salient objects in complex scenes and obtain better segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the salient target detection method of the present invention with mixed features superimposed on cross-attention. Implementation

[0016] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0017] The basic idea of ​​the present invention is to propose a target detection method based on hybrid feature superposition and cross attention, which can alleviate the problems of existing saliency detection models, such as insufficient ability to distinguish foreground and background in low-contrast scenes; lack of effective utilization of the complementary relationship between edge information and saliency information; and feature information offset caused by convolution operation affecting the integrity of detection results. Specifically, multi-level features are first extracted based on ResNet-50, and a cross attention module is introduced to effectively capture useful contextual information from distant pixels; under the premise of fully considering the complementarity of saliency detection and edge information, the multi-level features of saliency areas and edges are simultaneously refined by stacking multiple bidirectional information refinement units; finally, a U-shaped network is used to output the refined results.

[0018] See also Figure 1 , the present invention proposes a target detection method based on hybrid feature superposition cross attention refinement, the specific steps are as follows: A. Input the RGB image of the original image, the masked GT image, and automatically crop the resolution of the input image; Assume the input RGB image is , size is .

[0019] B. Input the residual neural network (Resnet) to obtain different levels of features of the RGB image; Specifically, the input residual neural network Resnet is divided into 5 stages, wherein the first stage is regarded as the preprocessing of the input image, and the last 4 stages are composed of bottlenecks. The present invention extracts the feature maps obtained in the last 4 stages. , The size is [\frac {H} {{2}^{i-1}},\frac {W} {{2}^{i-1}}] .

[0020] C. Use the cross attention mechanism to obtain the feature map of fused attention weights; Specifically, by calculating the vertical and horizontal feature weights of feature maps at different levels, three different sets of feature map outputs are obtained for each level. Then, the feature maps are concatenated in the channel dimension, and finally, a convolutional layer is connected, adding context information to the local feature maps, enhancing the local features, and capturing the information of the same row and column. To capture the information of all pixels in the entire image, the cross-attention module is applied twice to the feature maps at different levels to obtain dense context information for all pixels and obtain feature maps fused with attention weights.

[0021] D. Execute the multi-task strategy, and use two convolutional layers to perform independent feature extraction on the feature maps for salient object detection and edge detection; Let the feature maps of the saliency detection task and the edge detection task be respectively, where represents the output of the th level in step C, E. Execute the two-way refinement strategy, and perform multiple iterative refinements on the feature maps output at different levels for the two detection tasks in step D; Specifically, the feature maps are generated at four different levels. The features of the highest level of a certain detection task will only perform feature refinement with the features of the same level or higher levels of the other task, and will not use the features of lower levels than this level. For the saliency detection task, each two-way information refinement module executes the function, and for the edge detection task, each two-way information refinement module executes the function and are defined as follows: f = Conv(Cat({S}^{i}_{n - 1}, {Cat}^{4}_{k = i}[CU({E}^{k}_{n - 1})])) where, in represents the th level feature in the saliency feature input of the th two-way information refinement module. Among them, represents the concatenation (concat) operation on the channel axis, represents the convolution operation, is the scale conversion operation, which has 32 output channels at the same time, and the convolutional layer with a kernel size of . The purpose of this operation is to ensure that the spatial dimension sizes between the segmentation and edge features are consistent. When k > i, uses the bilinear upsampling operation with a scale factor of ; when k < i, uses the bilinear upsampling operation with a scale factor of Bilinear downsampling operation; when k=i, Use the identity function, in addition Represents pixel level multiplication.

[0022] F. Based on the feature map obtained by upsampling and concatenating features at different levels, a U-shaped network segmentation strategy is implemented to obtain the predicted saliency map and edge map; Specifically, by superimposing multiple bidirectional information refinement modules, we obtain multi-level features of two task refinements, and then use two typical U-Net decoding modules to upsample and cascade from high-level to low-level to generate two features of size The features of each upsampling and cascade operation are followed by a convolution operation. Through two convolution operations, two upsampling operations (scale factor 4) and one sigmoid function calculation, the predicted segmentation and edge map are obtained. .

[0023] G. Use binary cross entropy loss, use the true value graph and the predicted graph to calculate the loss function, and perform back propagation to correct the model parameters; Specifically, the loss function includes a saliency map loss function, an edge detection loss function, and a saliency map and edge detection weighted loss function, and three different loss functions are optimized and calculated.

[0024] The loss function is defined as follows: in It is the binary cross entropy loss, which is widely used in binary classification tasks and segmentation tasks. It is defined as follows: {L}_{BCE}=-\sum _{(x,y)} {[g(x,y)log(p(x,y))\, +\, (1-g(x,y))log(1-p(x,y))]} where g(x,y)\in [0,1] for At the point The label value at p(x,y)\in [0,1] for At the point The predicted value at .

[0025] H. Repeat steps B to G to revise the model until the set rounds are completed and output the final model.

Claims

1. A salient object detection method based on hybrid feature superposition and cross attention, characterized in that: The method comprises the following steps: A. Input the color image to be detected and the corresponding mask mark image and crop them to a uniform size; B. Use residual neural network to generate features of different levels corresponding to color images; C. Using the cross attention module, we generate feature maps based on saliency and edge-based multi-level fusion attention weights respectively; D. Execute a two-way refinement strategy to iteratively refine the features of different tasks at different levels in C; E. Execute the U-type prediction network to segment and predict the feature maps of different tasks to obtain the predicted edge map and saliency map; F. Output the final saliency segmentation map and saliency edge segmentation map.

2. The method according to claim 1, characterized in that: The feature extraction method described in step B is based on ResNet-50 to extract multi-level features of the RGB image.

3. The method according to claim 1, characterized in that: The cross-attention module described in step C is used to optimize the multi-level feature map. By adding the information of the same row and column pixel set to each pixel in the feature map, the contextual dependency between distant pixels in the feature map is obtained. The cross-attention module is cycled twice to capture the information of all pixels in the entire image. The feature maps of different levels are input into the cross-attention module respectively to generate feature maps with rich semantic information based on saliency detection and segmentation tasks.

4. The method according to claim 1, characterized in that: The bidirectional refinement strategy described in step D iteratively refines the features of different tasks at different levels. First, the complementarity between saliency information and edge information is utilized. For the feature map of the saliency detection task, the edge feature information is used to maintain the edge detail information. For the feature map of the edge detection task, the saliency information is used to obtain the positioning information. Secondly, the edge features (saliency features) of a certain level are fused with the saliency features (edge ​​features) of the same level and higher levels to maximize the use of the unclassified edge information in the low-level features and the high-level semantic information.

5. The method according to claim 1, characterized in that: The U-type prediction network described in step E performs segmentation prediction on the feature maps of different tasks to obtain predicted edge maps and saliency maps, and uses the decoding module in the U-Net network to generate features by upsampling and cascading (concat) operations from high layers to low layers. After convolution and upsampling operations and a sigmoid function calculation, the predicted saliency segmentation map and edge map are obtained.

6. The method according to claim 1, characterized in that: The network optimization model finds the best parameters by minimizing the loss function, which includes: saliency map loss function, edge detection loss function, and saliency map and edge detection weighted loss function. All three loss functions are based on the cross entropy loss function.

Citation Information

Cited By

  • Saliency detection and edge guidance optimization method based on multi-scale feature fusion

    CN119723273A