Camouflaged target image segmentation method based on omnidirectional perception

Through OPNet's all-round perception network, combined with local features and global representation, the accuracy problem of camouflage target segmentation in complex scenarios is solved, and the efficient camouflage target segmentation effect is achieved in different scenarios.

CN114549567BActive Publication Date: 2024-12-03DALIAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210167375.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-23
Publication Date
2024-12-03
Estimated Expiration
2042-02-23

AI Technical Summary

Technical Problem

The prior art is difficult to accurately segment the camouflage target in complex scenarios, especially due to the insufficient local features and global representation, resulting in inaccurate positioning and boundary delineation of the camouflage target.

Method used

A comprehensive perception network OPNet is proposed, combining Conformer extractor, pyramid positioning module and dual focus module to integrate local features and global representations. Through the pyramid self-attention mechanism and dual attention mechanism, precise positioning and boundaries of camouflage targets are achieved. Depict.

Benefits of technology

OPNet significantly improves the accuracy and efficiency of disguised target segmentation in different scenarios, can effectively process images of multi-scale and complex structures, and significantly improves the accuracy and performance of segmentation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114549567B_ABST
    Figure CN114549567B_ABST
Patent Text Reader

Abstract

The present invention belongs to the technical field of scene segmentation in computer vision, provides a camouflaged target image segmentation method based on omnidirectional perception, designs a new omnidirectional perception network for accurate camouflaged target segmentation, wherein a pyramid positioning module and a dual focus module are proposed to couple local features and global representations, so as to more accurately locate camouflaged targets and finely divide boundaries, thereby solving the problem of camouflaged target segmentation in computer vision. The method of the present invention camouflages the target segmentation result, and is effective and practical in different scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of scene segmentation in computer vision, realizes the segmentation of image content, and particularly relates to a method for segmenting camouflaged targets in real environment images. Background Art

[0002] Two-dimensional image segmentation refers to the technology of distinguishing pixels belonging to different objects in an image to determine information such as the size, shape and position of the target in the environment image. It is a key step from image processing to image analysis and has great application value. In recent years, scene segmentation-related tasks, such as object detection, depth estimation, salient region detection and shadow detection, have achieved significant performance improvements.

[0003] In nature, many organisms have evolved superb camouflage skills (e.g., protective coloration and mimicry), which can disguise themselves to blend into the surrounding environment and thus escape predators. Because of the high similarity between these organisms and the background, identifying camouflaged targets is a very challenging task for the human visual system. If such organisms appear in computer vision-based tasks, the accuracy of the task will be greatly affected. Therefore, camouflaged target segmentation is a very challenging task. At the same time, camouflaged target segmentation can help with tasks such as medical image analysis, locust invasion monitoring, and search and rescue, and it is also an indispensable task.

[0004] Early camouflaged target segmentation methods used low-level features such as edges, textures, and brightness to distinguish objects from the background. However, these methods are only applicable to some simple scenes and fail when faced with complex field scenes. Therefore, in recent years, research on camouflaged target segmentation often uses the powerful capabilities of deep neural networks to handle more challenging scenes. Inspired by the process of predators capturing prey in nature, many camouflaged target segmentation methods follow the pattern of first positioning and then focusing to detect the position, size, and shape of camouflaged targets. However, since these methods only make decisions based on regional-level contextual features, they are usually unable to locate camouflaged targets in more cluttered scenes, nor can they accurately outline the boundaries of camouflaged targets.

[0005] We have observed that top predators in nature usually have good central and peripheral vision at the same time, use the central vision to search for targets, and obtain clues from the scene environment through the peripheral vision. Therefore, the ideal camouflaged target segmentation model should be able to process local features and global representations at the same time when detecting targets. Based on the investigation and analysis of existing camouflaged target segmentation technologies, the present invention is inspired by the process of top predators capturing prey in nature, and proposes a new neural network to segment camouflaged targets by establishing an omnidirectional perception of the input scene, so that the model can use local features and global representations at the same time during the entire detection process, forming an omni-perception network (Omni Perception Network) for camouflaged targets in segmentation scenes, namely OPNet. The input of OPNet is an RGB image containing a camouflaged target, and the output is a mask image of the camouflaged target. The value of the camouflaged target area in the mask image is 1, and the value of the non-camouflaged target area is 0. Experiments have shown that this method can effectively segment different types of camouflaged targets in different scenes, helping to improve the accuracy of scene understanding. Summary of the invention

[0006] The purpose of the present invention is to innovatively use an omni-sensing method for the RGB image of the scene, integrate local features and global representations in detection, and then realize the segmentation of camouflaged targets. OPNet successively includes a feature extractor, a pyramid positioning module (Pyramid Positioning Module) and a dual focus module (Dual Focus Module), wherein the feature extractor uses Conformer as the backbone network to extract local features and global representations; the pyramid positioning module (Pyramid Positioning Module) combines local features and global representations to accurately locate target objects of different sizes; the dual focus module (Dyal Focus Module) combines local features and global representations to determine the boundaries of the target and eliminate interference backgrounds. The method described in the present invention is suitable for segmenting camouflaged targets of different types, positions and sizes.

[0007] The technical solution of the present invention:

[0008] A camouflaged target image segmentation method based on omnidirectional perception, the steps are as follows:

[0009] Step 1: Build the camouflage target segmentation network OPNet

[0010] The camouflaged object segmentation network OPNet mainly consists of a feature extractor, a pyramid localization module and three dual focus modules;

[0011] The backbone network of the feature extractor is Conformer, which extracts local features and global representations simultaneously through a dual network structure, namely the convolution branch and the Transformer branch, and interactively fuses local features and global representations through a functional coupling unit.

[0012] The pyramid localization module uses the pyramid self-attention mechanism to reduce the negative impact of shape changes and provide localization of camouflaged targets, thereby preliminarily determining the location of potential camouflaged targets in the image. Specifically, the input of the pyramid localization module is the local features C extracted from the convolutional branch of the Conformer and the Transformer branch and the global representation T (T includes N patch token embeddings T p and 1 segmentation token embeddingst s ). They are first converted to a consistent shape via adaptive average pooling and reshaping operations:

[0013]

[0014]

[0015] in,[] e represents the concatenation operation on the embedding dimension; ψ k Represents a k×k convolutional layer; Indicates adaptive average pooling with a target size of n×n; Represents a deformation operation; represents the regularization layer; Represents the activation function. Then, the two features are fused using element-wise addition:

[0016] F fusion =C down +T down ,

[0017] After this, F fusion Feedback to the joint dependency awareness module to detect global dependencies:

[0018]

[0019] in, Represents a multi-head self-attention mechanism; represents a multi-layer perceptron. The obtained F jdp It consists of two parts: N patch token embeddingsF p and 1 segmentation token embeddingsf sThen for F jdp Two types of transformations are performed to generate two types of features (for enhancing local features and global representation, respectively):

[0020]

[0021]

[0022] in, is the upsampling achieved by bilinear operation; given After the target size n in , the joint dependency perception module only performs global correlation perception at a fixed scale; at a specific scale, F fusion Each patch tokenembeddings in is a representation of a region of a specific size; by changing the joint dependency-aware module n to achieve multi-scale dependency perception:

[0023]

[0024]

[0025] in,[] c Represents the concatenation operation on the channel dimension; and is the output local feature and global representation of the joint dependency-aware module, where where n is equal to i; finally, the final output of the pyramid positioning module is generated as follows:

[0026]

[0027]

[0028] The next three dual-focus modules will further process C ppm and T ppm ;

[0029] The dual focus module is designed to help the network focus on the candidate region in the decoding stage. It uses local and global attention mechanisms to help complete the delineation of the boundary of the disguised object. It converts the local feature C output by the previous module into h , global representation E (including N patch token embeddings E p and 1 segmentation token embeddings e s ) and the current level local features C from the encoder cenc (like Figure 1 C cenc 1) as input. First, the higher-level local features C h And the global representation E performs channel focusing, and the joint vector required for channel focusing is generated in the following way:

[0030] V joint =[V c , V t ] c ,

[0031]

[0032]

[0033] Among them, V c and V t From C h and E p The generated focus vector; then the channel attention vectors of local features and global representation are obtained respectively, and these two vectors are used to enhance these two types of features:

[0034]

[0035] T cf =[e s , E′ p ] e ,

[0036]

[0037]

[0038]

[0039]

[0040] Among them, S is the activation function; and are used to enhance C combined and E p vector; C cf and T cf They are the local features and global representation of channel focusing respectively. Then, C cf and T cf Perform spatial focusing and obtain the spatial focus map:

[0041]

[0042] M c =ψ 7 (C h ),

[0043]

[0044]

[0045] in, is a token-to-graph generator; M c and M t From C h and the focal map generated by E; Γ is a linear projection that reduces the dimension of each global representation to 1 through a fully connected layer; d represents the length of a global representation; and represents three learnable linear embedding functions implemented by three fully connected layers; then further enhance C cf and T cf :

[0046]

[0047]

[0048]

[0049]

[0050] in, and They are used to enhance C cf and T cf Figure; C sf and T sf They are the local features and global representation of spatial focus respectively. Finally, the final output of the DFM module is generated, namely C dfm and T dfm :

[0051]

[0052]

[0053]

[0054]

[0055] in, represents MHSA followed by MLP with two residual connections to construct a sf Further exploring global dependencies; the “double” focusing module is constructed in two ways, i.e. it allows channel and spatial focusing, and focuses on local features and global representations; because local features have more detailed information, the output local features of the last DFM are finally used to predict the final output of OPNet:

[0056]

[0057] Step 2: Training process

[0058] During training, the training data of the dataset is first fed into the network’s feature extractor, and the extracted local features and global representations are fed into the pyramid localization module; the pyramid localization module establishes and enhances global dependencies, locates the camouflaged target in the local features and global representations, and then feeds the generated local features and global representations into three dual focus modules; the dual focus module uses local and global attention mechanisms to help complete the delineation of the camouflaged object boundaries, and ultimately obtains a refined prediction result. In order to make the network pay more attention to areas that may be predicted incorrectly, we use the weighted binary cross entropy loss (l wbce ) and weighted IoU loss (l wiou ) merged, that is To facilitate the training process, we introduce M in the three DFMs and the final output respectively. c and M t ; The overall loss function is defined as follows:

[0059]

[0060] Among them, α, β and γ are equilibrium constants, which are set to 1, 1 and 2, respectively, to guide OPNet to better predict the final camouflaged target segmentation result based on the input image.

[0061] Beneficial results of the present invention:

[0062] (1) Innovation of methods

[0063] This paper emphasizes the importance of both local features and global representation in camouflaged target segmentation, and introduces omnidirectional perception into the inherent camouflaged target segmentation paradigm of first positioning and then focusing. A new omnidirectional perception network (OPNet) for accurate camouflaged target segmentation is designed, in which two new modules (pyramid positioning module and dual focusing module) are proposed to couple local features and global representation in order to more accurately locate camouflaged targets and finely divide boundaries, solving the problem of camouflaged target segmentation in computer vision.

[0064] (2) Results and efficiency of camouflaged target segmentation

[0065] Our carefully designed pyramid localization module (PPM) can help the network better locate the target object from the perspective of multi-scale global dependency perception; our proposed dual focus module (DFM) can suppress interference with the help of global representation and obtain detailed information using local features, so OPNet can perform well in different scenarios. We use comparative experiments to prove the accuracy of OPNet's camouflaged target segmentation results. In the comparative experiments, OPNet outperforms in four common indicators (S α , The best performance was obtained on both M and OPNet, and the advantages were obvious. From the results, the segmentation results of OPNet are very good.

[0066] (3) Wider applicability

[0067] The present invention is applicable to the segmentation of various types of camouflaged targets. OPNet has obvious performance advantages on the following types of images: a) images containing small camouflaged targets; b) images containing large camouflaged targets; c) images containing camouflaged target areas with contents similar to the surrounding environment or unclear boundaries; d) images with camouflaged targets with complex structures. This proves the effectiveness and practicality of OPNet in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is the network structure of OPNet of the present invention.

[0069] Figure 2 The results of the embodiments of the present invention and the results of comparative experiments are shown. DETAILED DESCRIPTION

[0070] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0071] The datasets used in this embodiment are CHAMELEON (76 images), CAMO (1250 images), COD10K (5066 images) and NC4K (4121 images). We use 1000 images in CAMO and 3040 images in COD10K as training sets, and other images as test sets. During training, images of various sizes in the dataset will be uniformly scaled to a resolution of 416×416 and enhanced by random horizontal flipping and color jittering. The output results of image segmentation will be rescaled to the original size of the input image. The parameters of the encoder network are initialized by the Conformer-B model pre-trained on ImageNet, and the remaining layers are randomly initialized. The implementation of OPNet is based on PyTorch and uses a stochastic gradient descent optimizer with a momentum value of 0.9 and a weight decay of 5×10 -4We set the batch size to 14 and adjust the learning rate through the poly strategy. The baseline learning rate is 0.001 and the power is 0.9. The CPU of the running device is 72-core Intel XeonGold 6240 2.6GHz, the memory size is 32GB, and the GPU model is Tesla V100.

[0072] Figure 1 This is the network structure of OPNet. OPNet consists of a feature extractor, a pyramid localization module, and three dual focus modules. The feature extractor extracts local features and global representations simultaneously through a dual network structure, namely, a convolution branch and a Transformer branch, and interactively fuses local features and global representations through a functional coupling unit. The pyramid localization module combines local features and global representations to accurately locate target objects of different sizes; the dual focus module combines local features and global representations to determine the boundaries of the target and eliminate interference backgrounds, generating a mask image of the camouflaged target as the final camouflaged target segmentation result.

[0073] Figure 2 The results of the embodiments of the present invention and the comparative experimental results are shown in Figure 2. In order to verify the effectiveness of OPNet, we retrained other recent deep neural networks in the field for camouflaged target segmentation and used publicly available codes to train and test on the same dataset. The columns in the figure are the original image, manually annotated mask image, OPNet, SINet-V2, Li et al., R-MGL, Lvet al., PFNet, SINet, GCPANet, and CCNet from left to right.

Claims

1. A camouflaged target image segmentation method based on omnidirectional perception, It is characterized in that Here are the steps: Step 1: Build the camouflage target segmentation network OPNet The camouflaged object segmentation network OPNet mainly consists of a feature extractor, a pyramid localization module and three dual focus modules; The backbone network of the feature extractor is Conformer, which extracts local features and global representations simultaneously through a dual network structure, namely the convolution branch and the Transformer branch, and interactively fuses local features and global representations through a functional coupling unit. The pyramid localization module uses the pyramid self-attention mechanism. The input of the pyramid localization module is the local feature C extracted from the convolution branch and the Transformer branch of the Conformer and the global representation T, which includes N patch tokenembeddings T p and 1segmentation token embeddingst s ; First, the input is transformed into a consistent shape through adaptive average pooling and reshaping operations: in,[] e represents the concatenation operation on the embedding dimension; ψ k Represents a k×k convolutional layer; Indicates adaptive average pooling with a target size of n×n; Represents a deformation operation; represents the regularization layer; represents the activation function; then, the two features are fused using element-wise addition: F fusion =C down +T down , After this, F fusion Feedback to the joint dependency awareness module to detect global dependencies: in, Represents a multi-head self-attention mechanism; represents a multi-layer perceptron; The obtained F jdp It consists of two parts: N patch token embeddingsF p and 1segmentationtoken embeddingsf s ; then for F jdp Two types of transformations are performed to generate two types of features, which are used to enhance local features and global representation respectively: in, is the upsampling achieved by bilinear operation; given After the target size n in , the joint dependency perception module only performs global correlation perception at a fixed scale; at a specific scale, F fusion Each patch tokenembeddings in is a representation of a region of a specific size; by changing the joint dependency-aware module n to achieve multi-scale dependency perception: in,[] c Represents the concatenation operation on the channel dimension; and is the output local feature and global representation of the joint dependency-aware module, where where n is equal to i; finally, the final output of the pyramid positioning module is generated as follows: The next three dual-focus modules will further process C ppm and T ppm ; The dual focus module is designed to help the network focus on the candidate region in the decoding stage, using local and global attention mechanisms to help complete the delineation of the boundaries of the disguised object. It converts the local features C output by the previous module into h , the global representation E and the current level local features C from the encoder cenc As input, the global representation E contains N patch tokenembeddings E p and 1segmentation token embeddings e s ; First, the higher-level local features C h And the global representation E performs channel focusing, and the joint vector required for channel focusing is generated in the following way: V joint =[V c ,V t ] c , Among them, V c and V t From C h and E p The generated focus vector; then the channel attention vectors of local features and global representation are obtained respectively, and these two vectors are used to enhance these two types of features: T cf =[and s ,AND' p ] e , Among them, S is the activation function; and are used to enhance C combined and E p vector; C cf and T cf They are the local features and global representations of channel focusing respectively; then further cf and T cf Perform spatial focusing and obtain the spatial focus map: M c =ψ 7 (C h ), in, is a generator from token to graph; M c and M t From C h and the focal map generated by E; Γ is a linear projection that reduces the dimension of each global representation to 1 through a fully connected layer; d represents the length of a global representation; and represents three learnable linear embedding functions implemented by three fully connected layers; then further enhance C cf and T cf : in, and They are used to enhance C cf and T cf Figure; C sf and T sf They are the local features and global representation of spatial focus respectively; finally, the final output of the DFM module is generated, namely C dfm and T dfm : in, represents MHSA followed by MLP with two residual connections to construct a sf Further exploring global dependencies; the "double" focusing module is constructed in two ways, i.e. it allows channel and spatial focusing, and focuses on local features and global representations; because local features have more detailed information, the output local features of the last DFM are finally used to predict the final output of OPNet: Step 2: Training process During training, the training set data of the dataset is first fed into the network's feature extractor, and the extracted local features and global representations are fed into the pyramid localization module; the pyramid localization module establishes and enhances global dependencies, locates the camouflaged target in the local features and global representations, and then feeds the generated local features and global representations into three dual focus modules; the dual focus modules use local and global attention mechanisms to help complete the delineation of the boundaries of the camouflaged object, and ultimately obtain refined prediction results.

2. According to the camouflaged target image segmentation method based on omnidirectional perception according to claim 1, It is characterized in that In order to make the network pay more attention to the areas that may be predicted incorrectly, the weighted binary cross entropy loss l wbce And the weighted IoU loss l wiou Merge, that is To facilitate the training process, M is introduced in the three DFMs and the final output. c and M t ; The overall loss function is defined as follows: Among them, α, β and γ are equilibrium constants, which are set to 1, 1 and 2, respectively, to guide OPNet to better predict the final camouflaged target segmentation result based on the input image.