A camouflaged target detection method based on dual-branch feature fusion based on self-similarity

Abstract

A camouflaged target detection method based on self-similarity, using dual-branch feature fusion, involves multi-angle feature fusion, self-similarity constraints, and feature optimization. To address the problem that commonly used search-before-recognition network frameworks cannot effectively locate finer distinctions between foreground and background, the method: 1) uses a backbone to extract low-level, mid-level, and high-level features; 2) uses dual-branch feature fusion to obtain enhanced features for mid-level and high-level features; 3) further optimizes the features to obtain the final refined prediction map; 4) for the enhanced features, the mapped foreground and background regions are found based on the groundtruth, and their respective self-similarity matrices are calculated. For fine features, the groundtruth is directly used to impose global self-similarity constraints. Finally, the coarse prediction map, fine prediction map, and self-similarity loss are calculated. This method highlights finer distinctions between foreground and background, reducing the mean absolute error of the model detection.

Description

Technical Field

[0001] The present invention relates to self-similarity constraint, feature enhancement and optimization, and in particular to a camouflage target detection method based on self-similarity dual-branch feature fusion. Background Art

[0002] Organisms in nature adapt their features to changes in their environment to avoid detection, a process known as background matching camouflage. Similarly, camouflaged object detection, a specialized visual detection task, aims to locate the foreground within a similar background and foreground environment. Camouflaged object detection is a more challenging and challenging task than general object detection, but it has important applications, such as polyp segmentation in medicine and locust detection in agriculture. The primary difficulty and challenge of this task currently lies in the inherent similarity between the foreground and background, and further exploration is needed in the edge extraction process.

[0003] In recent years, the field of camouflaged object detection has advanced beyond the initial approaches based on handcrafted features. For example, researchers have leveraged various visual or other features, such as color, brightness, texture, and depth, to address various challenges. Subsequently, researchers have explored a new approach: deep learning. Various approaches have proposed various deep learning models based on convolutional neural networks to address the problem of camouflaged object detection. The paper (Deng-Ping Fan, et al. Camouflaged object detection [C]. / / In IEEE Conference on Computer Vision and Pattern Recognition. 2020: 2777–2787) constructed the largest dataset for camouflaged object detection to date in terms of both volume and variety, a significant step forward for further research. Furthermore, based on their dataset, they proposed the SINet network. This network consists of two main phases, modeled after the two stages of a predator's target search process. The first phase, called the search process, determines whether the target is within range. The results of the first phase are then supplemented by the second phase, which is called the recognition phase, which involves further determining the target's specific location. This paper provides a relatively general framework for camouflaged object detection, and many subsequent works have designed their own networks based on this model. The literature (Yan, J., Le, TN, Nguyen, KD, Tran, MT, Do, TT, Nguyen, TVMirrornet: Bio-inspired camouflaged object segmentation [C]. / / in IEEE Access. 2021: 43290–43300) introduces MirrorNet to use instance segmentation and mirror flow for camouflaged object segmentation. The literature (Jingjing Ren, et al. Deep texture-aware features for camouflaged object detection [J]. arXiv preprint arXiv: 2102.02996, 2021) uses a texture-aware refinement module to calculate the covariance matrix of feature responses to extract texture information.

[0004] In addition, there are many papers that add edge truth for guidance, so as to learn more edge information. Because the boundary between the camouflaged object and the background is quite fuzzy and difficult to distinguish, it is difficult to accurately locate the camouflaged object without introducing additional prior information. The paper (Yujia Sun, et al. Boundary-Guided Camouflaged Object Detection [J]. arXiv preprint arXiv: 2207.00794, 2022) proposes a boundary-guided module to learn boundary-enhanced representation to retain the local features and boundary information of the original image to improve the performance of the model. The paper (Hongwei Zhu, et al. I can find you Boundary-guided Separated Attention Network for Camouflaged Object Detection [C]. / / Proceedings of the AAAI Conference on Artificial Intelligence 36 (3). 2022: 3608-3616) proposes an edge-guided feature module (EFM) that aims to inject boundary-related edge cues into representation learning to enhance feature representation with object structure semantics. Compared to methods without edge ground truth, these methods are able to effectively detect more precise edges. However, without edge guidance, many issues remain to be optimized. Many methods lack high generalization capabilities when it comes to detecting small objects or finding more precise edges. Summary of the Invention

[0005] The purpose of the present invention is to address the problem that commonly used camouflaged target detection cannot well distinguish the fine differences between foreground and background without using additional edge truth values. A camouflaged target detection method based on self-similarity dual-branch feature fusion is provided. The method makes full use of the self-similarity between the foreground and background to better find information such as edges. On the one hand, the similarity between the foreground and background is widened, and on the other hand, the self-similarity of each is narrowed, providing a method for better distinguishing the foreground and background from the perspective of similarity. At the same time, the dual-branch feature fusion method is used to obtain more contextual information, thereby improving the generalization ability of the network.

[0006] The present invention comprises the following steps:

[0007] 1) Use backbone to extract features at different levels, including three groups of features: low-level, mid-level, and high-level;

[0008] 2) A dual-branch feature fusion approach is used to obtain enhanced features for mid- and high-level features. One branch uses dilated convolution to expand the receptive field, with a larger dilation ratio for feature fusion; the other branch uses multi-scale fusion to fuse multi-scale information from bottom-up and top-down.

[0009] 3) Use feature optimization to continue to obtain the final refined prediction map, and calculate the loss of the coarse prediction map and the refined prediction map;

[0010] 4) Apply self-similarity constraints to the features refined in step 3. Find the foreground and background regions after the enhanced feature mapping based on the ground truth, calculate their respective self-similarity matrices, and simultaneously calculate the losses for the coarse prediction map, fine prediction map, and self-similarity. Global supervision is performed on the prediction map and the ground truth, while supervision is performed on the foreground and background separately, increasing the similarity between the foreground and background regions while reducing the self-similarity of the foreground and background regions.

[0011] In step 1), the backbone is used to extract features of each layer. For a given input image Use Res2Net-50 to extract different layer features F i ,i∈[1,5], delete the pooling layer, fully connected layer and softmax, and finally obtain the feature layer containing high and low semantics, and divide it into low-level, medium and high-level features.

[0012] In step 2), in order to merge more distinctive information, a dual-branch feature fusion method is used to extract enhanced features. This module can extract features from different angles; specifically, it includes an SRF module and an MSSD module;

[0013] The SRF module consists of two parallel branches. It first increases the number of channels to 160 through a 1×1 convolution layer. Then, it is fed into two parallel branches. The first branch uses a 1×1 convolution operation to reduce the number of channels to 32. The other branch uses a dilated convolution method, using a 3×3 convolution with a dilation rate of 18. Finally, the two branches are merged and fed into the ReLu function at the same time to obtain the final output of the SRF branch.

[0014] In the second branch, to address the problem that a fixed receptive field is limited for smaller objects in multi-scale object detection, a structure such as MSSD is introduced to supplement the feature enhancement process. The features on F3, F4, and F5 are concatenated together, and then a new feature pyramid layer is generated by downsampling.

[0015] After obtaining the enhanced features, the feature decoder is used to obtain the coarse prediction map.

[0016] In step 3), the final fine prediction map is obtained, and the distinguishable hidden areas are mined by rerasing objects. Multi-stage refinement prediction is also performed step by step through different layers. The loss of the coarse and fine prediction maps uses weighted cross entropy and weighted IoU loss to supervise the network from both coarse and fine aspects.

[0017] In step 4), the mapped foreground and background regions are found based on the ground truth for the refined features, and subtle differences between the foreground and background are found based on the self-similarity limit. A self-similarity constraint module is proposed, which consists of two parts: the global constraints (GC) module and the Mutual constraints (MC) module.

[0018] In the GC module, for the refined prediction graph and ground truth, their self-similarity matrices are calculated respectively, and the mean square error is used for element-by-element supervision.

[0019] After obtaining the self-similarity constraint between the ground truth and the fine prediction map, the self-similarity of the foreground and background is also introduced for calculation, namely the MC module; specifically, after obtaining the coarse prediction map, the ground truth is used to find the mapping of the foreground and background, and the positive and negative similarities are calculated. The network is supervised by calculating the weighted similarity loss.

[0020] This invention addresses the problem that commonly used network frameworks that search first and then recognize are unable to effectively locate finer distinctions between foreground and background. By considering the value of the self-similarity between foreground and background, this invention makes full use of this information to reduce the distribution range of the foreground and background of the image. Specifically, it has the following outstanding advantages:

[0021] 1) In terms of early feature enhancement, the present invention uses dilated convolution to extract contextual features, which can increase the receptive field while obtaining more contextual information. In addition, compared with other methods that require four or five branches for aggregation, only two branches are used, and only the branches with a larger dilation rate are used, which can reduce the operation of redundant information.

[0022] 2) This invention uses self-similarity to address finer distinctions between foreground and background. Based on the refined feature maps, groundtruth is used to find feature maps for both the foreground and background. This approach not only narrows down the foreground and background features, but also separates them. This constraint allows for more detailed feature information to be found before subsequent feature optimization. Furthermore, global constraints are applied across the entire image.

[0023] 3) This invention employs a dual-branch feature enhancement approach. In addition to the aforementioned dilated convolution approach, the other branch fuses multi-scale information using bottom-up and top-down approaches, enhancing features from different perspectives. This invention also optimizes the detection of small objects, demonstrating the effectiveness of extracting features from different perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the main structural diagram of the present invention.

[0025] Figure 2 This is the GC module structure diagram of the SSCM module.

[0026] Figure 3 This is the MC module structure diagram of the SSCM module.

[0027] Figure 4 Visualize the results for the test set. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the following embodiments will be further described in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention. On the contrary, the present invention encompasses any substitutions, modifications, equivalent methods and solutions made within the spirit and scope of the present invention as defined by the claims. Furthermore, in order to provide the public with a better understanding of the present invention, certain specific details are described in detail in the following detailed description of the present invention. Those skilled in the art can fully understand the present invention without the description of these details.

[0029] The purpose of the present invention is to address the problem that general camouflaged target detection methods have slightly poor detection effects on camouflaged targets, including small targets, without boundary truth supervision, and explore the subtle differences between foreground and background from the perspective of self-similarity.

[0030] The SimBNet network is proposed. It first obtains enhanced features through early feature enhancement, and then obtains a refined prediction map by means of feature refinement. During this process, the groundtruth and the refined features are self-similarly constrained, and similarity constraints are also imposed on the foreground and background. In addition, a dual-branch approach is used in the feature enhancement part to simulate the way humans capture different information from different observation angles when observing camouflaged targets. Similarly, the dual branches also extract more contextual information from different angles, which will reduce information loss. Specifically, input an image Then use the backbone network to extract multi-scale features, denoted as F i,i∈[1,5]. Then, in order to simulate the behavior of humans searching for targets from multiple angles, the extracted features are sent to the dual branches for feature enhancement, such as Figure 1 The left half of the image extracts more contextual information from different angles, which is beneficial for detecting small-scale targets. The first branch utilizes dilated convolution to increase the receptive field. This SRF module is a simplified version of the RF module, aiming to expand the receptive field while reducing redundant information. The second branch directly processes the three feature layers F3, F4, and F5 from bottom to top and top to bottom, integrating more information and capturing more target clues. The feature refinement module is then used to obtain the final prediction map. To preserve more detailed distinguishing information such as edges, a self-similarity module is used without introducing other supervisory conditions. This module leverages the self-similarity of the foreground to identify the extreme differences from the background. This learning method allows for the extraction of more refined information.

[0031] The specific process is as follows:

[0032] 1. Feature extraction

[0033] For a given input image Use Res2Net-50 to extract different layer features F i ,i∈[1,5], and divide it into low-level, medium-level and high-level features, where the pooling layer, fully connected layer and softmax are deleted, so that a feature layer containing high and low semantics is obtained.

[0034] 2. Dual-branch Feature Enhancement Module (BFEM)

[0035] To incorporate more discriminative information, a bibranched feature enhancement module (BFEM) is designed to extract features from different angles. The BFEM mainly consists of the Simple Receptive Field (SRF) module and the Model Single Shot Detector (MSSD) module.

[0036] The SRF module consists of two parallel branches. First, a 1×1 convolutional layer increases the number of channels to 160. This is then fed into two branches. The first branch uses a 1×1 convolution to reduce the number of channels to 32, while the other branch uses a dilated convolution, a 3×3 convolution with a dilation ratio of 18. Finally, the two branches are merged and fed into a ReLu function to obtain the output of the SRF branch.

[0037] In the second branch, in order to address the problem that fixed receptive fields are limited for smaller targets in multi-scale target detection, a structure such as MSSD is introduced to supplement the feature enhancement process. The features on F3, F4, and F5 are concatenated together, and then a new feature pyramid layer is generated by downsampling.

[0038] The entire BFEM module is as follows Figure 1 Shown in the left half.

[0039] 3. Feature decoder module

[0040] Like the classic method SINet-V2, low-level features perform poorly but consume more resources, so the last three layers of features are also used and sent to the feature decoder to obtain a coarse prediction map.

[0041] The recognition stage also uses their principled strategy to mine discriminative hidden regions by rerasing objects. In addition, multi-stage refinement can improve performance. Therefore, they also gradually refine the predictions at different layers, and this module is collectively referred to as GandR.

[0042] 4. Self-Similarity Module (SSCM)

[0043] After obtaining the refined prediction image, self-similarity is introduced to compensate for the loss of finer information, such as edges, to find more subtle differences between the foreground and background. The Self-Similarity Constraints Module consists of two parts: the Global Constraints (GC) module and the Mutual Constraints (MC) module.

[0044] Figure 2 The GC module structure is given in the figure. In the GC module, the fine prediction map is denoted as S3, which is then divided into blocks w of size 22×22, and then normalized, that is:

[0045]

[0046] Then calculate the self-similarity matrix of the features:

[0047]

[0048] Similarly, for the ground truth, its similarity matrix Sim_GT is also calculated, and then the mean square error is used to perform element-by-element supervision.

[0049] After obtaining the self-similarity constraint between the ground truth and the predicted image, the similarity of the foreground and background images is also introduced for calculation, which is the second part of the MC module. Figure 3 The MC module structure diagram is given. Compared with the first part that makes overall constraints, individual constraints are also introduced. Specifically, after obtaining the rough prediction image S5, the ground truth is used to find the mapping of the foreground and background, and the foreground is recorded as F for , the background is denoted as F bac ,in

[0050] F for =S5×GT threshold>0.01

[0051] F bac =S5×GT threshold＝0

[0052] Then calculate the similarity of the foreground and background:

[0053] Sim_FOR=F for ×F for T

[0054] Sim_BK=F bac ×F for T

[0055] Then the positive similarity is defined as:

[0056]

[0057] Define negative similarity as:

[0058]

[0059] Among them, θ represents the temperature coefficient, which is set to 0.07 here, and K represents the number of features for all similarity calculations. The final similarity loss is:

[0060] l sim_l =l eng +λ×l pos

[0061] Among them, λ = 0.01. This module supervises the whole image from the perspective of self-similarity on the one hand, and supervises from the perspective of foreground and background on the other hand, which is conducive to finding edge information with smaller differences between foreground and background.

[0062] 5. Loss Function

[0063] Because the edges between the foreground and background are difficult to distinguish and small targets are prone to information loss, the present invention integrates the losses of five parts from the perspective of the loss function, including the loss of the coarse prediction graphs of the two branches in the early dual-branch feature enhancement. coa and l init , using weighted cross entropy and weighted IoU loss, followed by the loss l of the final refined prediction map fin , and the two parts of the loss mentioned in the self-similarity module, the first part is the mean square error loss l of the overall self-similarity sim_g , the other part is the similarity loss l sim_l , the present invention makes different constraints on the above-mentioned losses, and the final loss is defined as:

[0064] l all =l oca +l init +l fin +l sim_g +l sim_l

[0065] The specific experimental results are as follows:

[0066] Three widely used COD benchmark datasets are used to test each method. COD10K contains 3040 training images and 2026 test images. CAMO includes 1000 training images and 250 test images, while CHAMELEON consists of 76 images. The training set consists of 3040 COD10K training images (3040 images), 600 CPD1K training images, and 1000 CAMO training images. The results of different COD methods are tested on the test set of COD10K (COD10K-Test), the test set of CAMO (CAMO-Test), and the entire CHAMELEON.

[0067] Currently, the mean absolute error (MAE) is widely used in COD tasks to evaluate the pixel-level accuracy between the predicted image and the true value. In addition, the E measurement (E) proposed in the literature (Deng-Ping Fan, Ge-Peng Ji, Guolei Sun, Ming-MingCheng, Jianbing Shen, and Ling Shao. Camouflaged object detection [C]. / / In IEEE Conference on Computer Vision and Pattern Recognition. 2020: 2777–2787) is also used. Φ ), this method can evaluate pixel-level matching and image-level statistics simultaneously. In addition, using the s-measure (Sα ) as a structural similarity evaluation metric. Weighted Fmeasty It can also provide more reliable evaluation results.

[0068] To verify the effectiveness of our method, we used the same training settings as the baseline. We optimized the network using stochastic gradient descent (SGD) and a poly learning strategy, with an initial learning rate of 0.0001 and a decay rate of 0.1. Training was terminated after 100 epochs. The input image was resized to 352×352.

[0069] The model was evaluated on the entire CHAMELEON dataset and the test sets of CAMO and COD10K. The specific experimental results are shown in Table 1, where M represents the MAE metric, the smaller the better, and the higher the other metrics, the better.

[0070] Table 1

[0071]

[0072] As can be seen from Table 1, the present invention achieves better results in multiple indicators without using the true edge value. Φ Compared with the SINet version, it is improved by 10.2%. α 5.8% increase, The improvement is 23.9%, and the mean absolute error (MAE) is reduced by 1.6%. And from the visualization results, Figure 4 It can also be seen that the present invention is more accurate in detecting disguised targets, multiple targets can also be effectively detected, and the generalization ability of the overall network is improved.

Claims

1. A camouflaged target detection method based on self-similarity dual-branch feature fusion, characterized by The following steps are involved: 1) Use backbone to extract features at different levels, including three groups of features: low-level, mid-level, and high-level; 2) A dual-branch feature fusion approach is used to obtain enhanced features for mid- and high-level features. One branch uses dilated convolution to expand the receptive field, with a larger dilation ratio for feature fusion; the other branch uses multi-scale fusion to fuse multi-scale information from bottom-up and top-down. In order to merge more distinguishable information, a dual-branch feature fusion method is used to extract enhanced features. This module extracts features from different angles; specifically, it includes the SRF module and the MSSD module. The SRF module consists of two parallel branches. It first increases the number of channels to 160 through a 1×1 convolution layer. Then, it is fed into two parallel branches. The first branch uses a 1×1 convolution operation to reduce the number of channels to 32. The other branch uses a dilated convolution method, using a 3×3 convolution with a dilation rate of 18. Finally, the two branches are merged and fed into the ReLu function at the same time to obtain the final output of the SRF branch. In the second branch, to address the problem that a fixed receptive field is limited for smaller objects in multi-scale object detection, a structure such as MSSD is introduced to supplement the feature enhancement process. The features on F3, F4, and F5 are concatenated together, and then a new feature pyramid layer is generated by downsampling. After obtaining the enhanced features, the feature decoder is used to obtain the coarse prediction map; 3) Use feature optimization to continue to obtain the final refined prediction map, and calculate the loss of the coarse prediction map and the refined prediction map; 4) Apply self-similarity constraints to the features refined in step 3), find the foreground and background regions after the enhanced feature mapping based on the ground truth, calculate their respective self-similarity matrices, and simultaneously calculate the losses of the coarse prediction map, fine prediction map, and self-similarity.

2. A method for detecting camouflaged targets based on dual-branch feature fusion based on self-similarity as claimed in claim 1, characterized in that In step 1), the backbone is used to extract features of different layers. For a given input image Use Res2Net-50 to extract different layer features F i ,i∈[1,5], delete the pooling layer, fully connected layer and softmax, and finally obtain the feature layer containing high and low semantics, and divide it into low-level, medium and high-level features.

3. A method for detecting camouflaged targets based on dual-branch feature fusion based on self-similarity as claimed in claim 1, characterized in that In step 3), the final fine prediction map is obtained, and the distinguished hidden areas are mined by erasing the objects, and multi-stage refined predictions are also performed step by step through different layers; the loss of the coarse and fine prediction map uses weighted cross entropy and weighted IoU loss to supervise the network from both coarse and fine aspects.

4. A method for detecting camouflaged targets based on self-similarity dual-branch feature fusion as claimed in claim 1, characterized in that In step 4), the mapped foreground and background regions are found based on the ground truth for the refined features, and the subtle differences between the foreground and background are found based on the self-similarity limit. A self-similarity module is proposed, which consists of two parts, namely the GC module and the MC module. In the GC module, for the refined prediction graph and groundtruth, their self-similarity matrices are calculated respectively, and the mean square error is used for element-by-element supervision. After obtaining the self-similarity constraint between the ground truth and the fine prediction map, the self-similarity of the foreground and background is also introduced for calculation, namely the MC module; specifically, after obtaining the coarse prediction map, the ground truth is used to find the mapping of the foreground and background, and the positive and negative similarities are calculated. The network is supervised by calculating the weighted similarity loss.

Images