A method and system for directional camouflage object segmentation

Abstract

This invention proposes a method and system for segmenting directional camouflaged objects. A dual-branch network structure is designed, utilizing a reference branch network to acquire common information about a certain type of directional object in a reference image. The segmentation branch network, guided by this common information, identifies and segments the target image containing the directional camouflaged object. The method highlights the directional camouflaged object in the target image by interactively fusing the common information and the visual features of the target image. This method is less affected by interference from other objects, thus enabling more accurate segmentation of the desired object. Furthermore, the method of highlighting directional camouflaged objects in the target image using common information from the reference image is suitable for directional segmentation of a certain type of object using reference information collected from a small sample size.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method and system for segmenting directional camouflage objects. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Camouflaged object detection (COD) aims to segment objects that are visually hidden in their surroundings. Due to the high similarity between camouflaged objects and their surrounding background, the diversity in size of camouflaged objects, and the ambiguity in their appearance, camouflaged object detection is more challenging than traditional object detection. One early systematic study was presented by Dengping Fan et al. in their paper "Concealed Object Detection," which released a large-scale dataset COD10K with high-quality annotations and a well-designed search and confirmation framework, SINet. Subsequently, numerous strategies, such as multi-scale features, multi-stage optimization, and joint training, have been proposed to improve the performance of camouflaged object detection models. Furthermore, recent work has introduced additional information, such as boundaries, texture, frequency domain, and depth, as guidance to further improve the accuracy of segmentation results. Although the above methods have achieved excellent performance on camouflaged object detection tasks, they all indiscriminately segment all possible camouflaged objects, without specifically telling the model what to segment.

[0004] Directed object segmentation means segmenting a specified object from an image under the guidance of some form of guidance information (e.g., image, text, etc.). Few-shot segmentation (FSS) explores object segmentation guided by images containing objects of the same class. In this study, the model is trained on a large number of samples labeled with a baseline class (query set) and performs dense pixel predictions given a small number of samples labeled with a new class (support set). Most existing few-shot segmentation networks contain two branches, a query branch and a support branch, to extract features from the query image and support images respectively and to enable interaction between them. The pioneering work in this field was proposed by A. Mirrezha Shaban et al. in "One-shot learning for semanticsegmentation," where the support branch directly predicts the weights of the last network layer of the query branch to segment the target object. Subsequently, Xiaolin Zhang et al. proposed an occluded average pooling operation in "Sg-one: Similarity guidance network for one-shot 1048semanticsegmentation" to extract representative support features, a technique that has been widely adopted in subsequent works. Recently, a great deal of work has focused on building powerful modules on fixed-parameter foundational networks to improve the model’s ability to adapt to new categories.

[0005] Referring Expression Segmentation (RES) explores object segmentation guided by textual expressions. In this research, most networks also employ a two-branch structure. Ronghang Hu et al. first proposed this approach in their paper "Segmentation from natural language expressions," where visual and linguistic features were extracted by a visual encoder and a linguistic encoder, respectively, and their concatenation was used to generate a segmentation mask. Subsequently, multi-level visual features, multimodal LSTM, attention mechanisms, and collaborative networks have been introduced into referring expression segmentation methods to generate more accurate results. Furthermore, Xiaoshuai Sun et al., in their paper "Exploring language prior for mode-sensitive visual attention modeling," also used textual expressions as a reference for image content richness to achieve better human gaze prediction.

[0006] The biggest challenge in applying the aforementioned directional object segmentation methods to camouflage object detection is the high time and resource costs involved in obtaining high-quality reference information from existing camouflage object detection datasets. For few-shot segmentation methods, collecting images containing camouflage objects of the same category and annotating them at the pixel level is difficult; for directional expression segmentation methods, accurately annotating each image containing camouflage objects with textual descriptions is also time-consuming and laborious. Therefore, there is an urgent need for a directional camouflage object detection method that can achieve directional segmentation of certain objects based on some low-cost collected reference information. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention provides a method and system for directional camouflage object segmentation, which uses common information of reference images to highlight directional camouflage objects in the target image. This method is suitable for directional segmentation of a certain type of object using reference information collected from a small sample.

[0008] To achieve the above objectives, a first aspect of the present invention provides a method for segmenting directional camouflage objects, comprising:

[0009] Feature extraction is performed on multiple reference images of the directional camouflage object based on a reference branch network to obtain the common characteristics of the directional camouflage object;

[0010] The target image, including the directional camouflage object, is input into a trained segmentation branch network for segmentation.

[0011] The process of segmenting the target image using a segmentation branch network is as follows: extracting the first visual features from the target image including the directional camouflage object;

[0012] The extracted first visual features and the common representation of the directional camouflage object are fused to obtain fused features, and a reference mask is generated based on the fused features and the common representation.

[0013] Based on the fusion features and the reference mask, the segmentation result of the target image is obtained.

[0014] A second aspect of the present invention provides a directional camouflage object segmentation system, comprising:

[0015] Reference branch network unit: used to extract features from multiple reference images of the directional camouflage object based on the reference branch network, and obtain the common representation of the directional camouflage object;

[0016] Segmentation branch network unit: used to input the target image including the directional camouflage object into the trained segmentation branch network for segmentation; the segmentation branch network includes an extraction unit, a fusion unit, a reference mask generation unit, and a segmentation unit;

[0017] Extraction unit: used to extract first visual features from the target image including the directional camouflage object;

[0018] The fusion unit is used to fuse the extracted first visual features and the common representations of the directional camouflage objects to obtain fused features;

[0019] A reference mask generation unit is used to generate a reference mask based on the fusion features and the common representations;

[0020] A segmentation unit is used to obtain a segmentation result of the target image based on the fusion features and the reference mask.

[0021] A third aspect of the present invention provides a computer device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, a directional camouflage object segmentation method is performed.

[0022] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs a method for segmenting directional camouflaged objects.

[0023] The above one or more technical solutions have the following beneficial effects:

[0024] In this invention, a dual-branch network structure is designed. A reference branch network is used to acquire common information about a certain type of directional object in a reference image, while a segmentation branch network, guided by the obtained common information, identifies and segments the target image containing the directional camouflage object. The directional camouflage object in the target image is highlighted by interactively fusing the common information and the visual features of the target image. This method is less affected by interference from other objects, thus enabling more accurate segmentation of the desired object. Furthermore, the method of highlighting directional camouflage objects in the target image using common information from the reference image is suitable for directional segmentation of a certain type of object using reference information collected from a small sample size.

[0025] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0027] Figure 1 This is a statistical and sample diagram of the directional camouflage object detection dataset constructed in Embodiment 1 of the present invention;

[0028] Figure 2 This is a schematic diagram of a directional camouflage object segmentation framework based on a dual-branch network structure in Embodiment 1 of the present invention;

[0029] Figure 3 This is a schematic diagram of the reference mask generation module in Embodiment 1 of the present invention;

[0030] Figure 4 This is a schematic diagram illustrating how the performance of the directional camouflage object segmentation method proposed in Embodiment 1 of the present invention varies with the number of reference images;

[0031] Figure 5 This is a schematic diagram comparing intermediate features of the directional camouflage object segmentation method proposed in Embodiment 1 of the present invention with and without reference information;

[0032] Figure 6 This is a schematic diagram illustrating the segmentation effect of the directional camouflage object segmentation method proposed in Embodiment 1 of the present invention and the traditional camouflage object detection method on camouflage targets. Detailed Implementation

[0033] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0034] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0035] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0036] Example 1

[0037] like Figure 2 As shown, this embodiment discloses a method for segmenting directional camouflage objects, including:

[0038] Step 1: Extract features from multiple reference images of the directional camouflage object based on a reference branch network to obtain the common characteristics of the directional camouflage object;

[0039] Step 2: Input the target image including the directional camouflage object into the trained segmentation branch network for segmentation;

[0040] The process of segmenting the target image based on the segmentation branch network is as follows: extract the first visual features from the target image including the directional camouflage object;

[0041] The extracted first visual features and the common representation of the directional camouflage object are fused to obtain fused features, and a reference mask is generated based on the fused features and the common representation.

[0042] Based on the fusion features and the reference mask, the segmentation result of the target image is obtained.

[0043] like Figure 1 As shown, in this embodiment, a dataset containing two subsets is constructed, and a model framework with a dual-branch network structure is proposed. In this dataset, the two subsets are the segmentation subset and the reference subset. The former consists of images containing camouflaged objects, and the latter consists of images containing similar objects and simple scenes. In this model framework, the two branches are the reference branch and the segmentation branch. The reference branch is used to capture the common representation of a certain type of object in the reference image, and the segmentation branch segments the camouflaged target object based on this common representation.

[0044] In this embodiment, we investigated the datasets widely used in current research on disguised objects detection, and determined which categories of disguised objects to be segmented based on the COD10K dataset, which has the largest data size and most comprehensive annotations. After removing a few uncommon categories, we obtained a segmentation subset consisting of 4966 images containing disguised objects, covering 64 disguise categories.

[0045] For images containing one or more camouflaged objects, the mask annotations provided by COD10K are used directly; for images containing multiple categories of camouflaged objects, the pixels corresponding to objects other than the reference category in the mask annotations provided by COD10K are removed.

[0046] A reference subset is constructed, consisting of images containing similar objects and simple scenes. This embodiment uses 64 categories from the segmented subset as keywords, collecting images from websites such as Flickr and Unsplash that are free of copyright issues and contain real-world scenes. For each category, 25 images containing a single object and a real-world scene are collected.

[0047] Furthermore, in this embodiment, for the reference subset, 20 images are randomly selected from the 25 images in each category for model training, and the remaining 5 images are used for model testing. For the split subset, the setup of the COD10K dataset is followed, that is, images belonging to the training set in each category are used for model training, and images belonging to the test set are used for model testing.

[0048] In step 1 of this embodiment, the reference branch network consists of a foreground prediction network based on an encoder-decoder structure and an occlusion average pooling function. Given several reference images, the encoder module of the foreground prediction network extracts the visual features of these images, its decoder module converts the visual features into a predicted foreground image mask, and the occlusion average pooling function extracts the common representation of a certain category of objects in the reference images based on the obtained visual features and the foreground image mask. The foreground prediction network used in this embodiment can be any currently available mainstream foreground prediction method, and the parameters of the foreground prediction network remain unchanged during the training and inference processes of the model.

[0049] The specific operation process of the reference branch network on the reference image is as follows: the encoder of the trained foreground prediction network is used to encode multiple reference images to obtain the second visual features corresponding to the reference images;

[0050] The second visual features are input into the decoder of the trained foreground prediction network to decode and predict the foreground image of the object in the reference image.

[0051] The predicted foreground image is used to mask the second visual features, filtering out information other than objects of a certain category;

[0052] The occluded second visual features are subjected to average pooling to obtain the semantic vector representation of the object category.

[0053] The common representation of objects of this category is obtained by averaging the semantic vector representations of multiple reference images.

[0054] Specifically, K reference images The reference branch fed into the framework generates the corresponding second visual features. and foreground image The generated foreground image is then used as a mask to filter out information other than the desired object from the features, obtaining the semantic vector representation of the object. This calculation process can be represented as:

[0055]

[0056] in, ∑ represents the downsampling operation used for shape matching. 2d (·) represents the summation of all eigenvalues ​​along the spatial dimension. This represents a 1×1 convolution operation used to transform the channel dimension.

[0057] In step 2 of this embodiment, the segmentation branch network consists of an encoder for visual feature extraction, a Referring Mask Generation (RMG) module, a Referring Feature Enhancement (RFE) module, and a decoder for segmenting the camouflaged target. Specifically, the encoder uses the most widely used feature extractor, namely a 50-layer residual network (ResNet50); the decoder consists of a convolutional block containing two convolutional layers, where the first layer has a 3×3 kernel size and the second layer has a 1×1 kernel size.

[0058] like Figure 3 As shown, the reference mask generation module includes a spatial information injection component consisting of three parallel feature concatenation operations, a cross-source fusion component consisting of two multilayer linear perceptrons and three parallel convolutional blocks with a kernel size of 3×3, a cross-scale fusion component consisting of two convolutional long short-term memory (Conv-LSTM) blocks, and a dense contrast module based on convolutional operations.

[0059] The common category representation E of objects of this category is obtained by calculating the mean of the K semantic vector representations in step 1. cate and multi-layered first visual features The data is fed into the Reference Mask Generation (RMG) module to generate the reference mask E. cate .

[0060] In the RMG module, considering that common category features and multi-level visual features come from different image sources, and that camouflaged objects and single objects of the same category may differ significantly in appearance, there is a large information difference between common category features and multi-level visual features. Furthermore, obtaining the reference mask by directly comparing them at the pixel level may interfere with the segmentation process. To address this issue, this example fuses the visual features containing the camouflaged object and the visual features from the reference image in the RMG module to eliminate the information difference between them.

[0061] Specifically, there are visual features ranging from high to low scale. The spatial information is input into the RMG module's spatial information injection component, based on the first visual features. The size encodes an 8-dimensional embedding vector for each position and incorporates visual features. Visual features are obtained by concatenating them with their corresponding spatial location codes. This allows spatial location information to be injected into visual features, thereby promoting the fusion of different representational information in the future.

[0062] Next, the cross-source fusion component uses affine transformation to achieve information fusion between the visual feature and the common representation of the reference object. Specifically, the category common representation E cate Two vectors are obtained by passing the data through two multilayer linear perceptrons. These two vectors are then compared with visual features containing spatial location information. The three features are fused sequentially using a linear affine transformation. The fused features are then processed by three parallel convolutional blocks to obtain features Y1, Y2, and Y3.

[0063] Subsequently, the cross-scale fusion component fuses the three cross-source fusion features Y1, Y2, and Y3 from different feature pyramid scales. This fusion process effectively eliminates the differences between the two-source representations. Specifically, Y1 and Y2 are input into the first convolutional long short-term memory (LSTM) module, and the output of the first LTM module, along with Y3, is input into the second LTM module. The second LTM module outputs the fusion result E of the category commonality representation and visual features. f .

[0064] Based on this, densely compared components characterize E using category commonality. cate For convolution kernels, dual-source fusion features E f Using the input features, the common characteristics of the categories E are represented through convolution operations. cate With fusion feature E f Each pixel is compared at the pixel level to generate a reference mask. This process can be represented as:

[0065]

[0066] Wherein, RMG(·) is the reference mask generation module proposed in this paper, and E f The result is a fusion of common category representations and visual features. This is the generated reference mask.

[0067] like Figure 2 As shown, the reference feature enhancement module includes a reference guidance component consisting of average pooling, resizing, and concatenation operations, and a feature enhancement component consisting of three parallel 3×3 convolutional blocks. Given an image I containing camouflaged objects... c ∈R H×W×3 The image is encoded by the encoder module, generating visual features at three feature pyramid levels. We use 1×1 convolutions to process these three feature pyramid levels to unify the channel dimensions and represent them as follows:

[0068] The fusion feature E generated abovef and reference mask The input is fed into the Reference Feature Enhancement (RFE) module to generate enhanced visual features. In the RFE module, average pooling is first applied to the dual-source fused features E... f The dimensions are adjusted to match the visual features of the three feature pyramids used in the reference mask generation. The scale of the reference mask is adjusted by resizing the reference mask. They were also adjusted to these three scales respectively. The scale is then determined; next, reference masks and fused features of the same scale are concatenated to enhance the ability to identify disguised objects; to improve the model's ability to recognize disguised objects at different scales, a path from high scale to low scale is constructed, sequentially concatenating higher-scale features with adjacent lower-scale features to achieve multi-scale visual feature interaction. Furthermore, this example also supervises the output features of the three scale paths, making the final output features more robust. Under the multi-scale guidance of the reference mask, the fused features are enhanced, thereby highlighting the disguised target object. This process can be represented as:

[0069]

[0070] Wherein, RFE(·) is the reference feature enhancement module proposed in this paper, E enr Visual features enhanced using a reference mask.

[0071] Finally, the enhanced features are fed into the decoder of the segmentation branch to generate the segmentation mask.

[0072] This embodiment conducted extensive experiments to verify the effectiveness of the directional camouflage object detection method. Figure 4 The diagram illustrates how the performance of the method in this embodiment changes with the number of reference images. A larger wF value indicates better method performance, and a smaller M value also indicates better method performance. It can be seen that the performance of the method in this embodiment is positively correlated with the number of reference images. This is because as the number of reference images increases, the influence of differences in the appearance of objects within that category on the obtained category's commonality representation decreases, thus enabling more accurate localization and segmentation of camouflaged targets. Figure 5 The diagram shows a comparison of intermediate features with and without reference information. Figure 6 This is a schematic diagram showing the segmentation effect of the method in this embodiment and the traditional camouflage object detection method on the camouflage target. It can be seen that, guided by the reference information, on the one hand, the method in this embodiment is less affected by interference from other objects, and thus more accurately segments the desired object; on the other hand, the method in this embodiment performs better in segmenting the main body of the camouflage object.

[0073] Example 2

[0074] The purpose of this embodiment is to provide a directional camouflage object segmentation system, including:

[0075] Reference branch network unit: used to extract features from multiple reference images of the directional camouflage object based on the reference branch network, and obtain the common representation of the directional camouflage object;

[0076] Segmentation branch network unit: used to input the target image including the directional camouflage object into the trained segmentation branch network for segmentation; the segmentation branch network includes an extraction unit, a fusion unit, a reference mask generation unit, and a segmentation unit;

[0077] Extraction unit: used to extract first visual features from the target image including the directional camouflage object;

[0078] The fusion unit is used to fuse the extracted first visual features and the common representations of the directional camouflage objects to obtain fused features;

[0079] A reference mask generation unit is used to generate a reference mask based on the fusion features and the common representations;

[0080] A segmentation unit is used to obtain a segmentation result of the target image based on the fusion features and the reference mask.

[0081] Example 3

[0082] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the aforementioned directional camouflage object segmentation method.

[0083] Example 4

[0084] The purpose of this embodiment is to provide a computer-readable storage medium.

[0085] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the directional camouflage object segmentation method described above.

[0086] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0087] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0088] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for segmenting directional camouflaged objects, characterized in that, include: Feature extraction is performed on multiple reference images of the directional camouflage object based on a reference branch network to obtain the common characteristics of the directional camouflage object, specifically: The encoder of the reference branch network is used to encode multiple reference images to obtain the corresponding second visual features; The second visual feature is input into the decoder of the reference branch network to obtain the predicted foreground map of the directional camouflage object; The predicted foreground image is used to mask the second visual feature; Average pooling is performed on the masked second visual features to obtain the semantic vector representation of the directional camouflaged object. The common representation of directional camouflaged objects is obtained by averaging the semantic vector representations of directional camouflaged objects in multiple reference images. The target image, including the directional camouflage object, is input into a trained segmentation branch network for segmentation. The process of segmenting the target image using a segmentation branch network is as follows: extracting first visual features from the target image including the directional camouflage object; The extracted first visual features and the common representation of the directional camouflage object are fused to obtain fused features. A reference mask is generated based on the fused features and the common representation. Specifically, the fused features are used as input and the common representation is used as the convolution kernel by using dynamic convolution. The reference mask is generated by comparing each pixel of the common representation and the fused features. Based on the fusion features and the reference mask, the segmentation result of the target image is obtained.

2. The method for segmenting directional camouflage objects as described in claim 1, characterized in that, The encoder of the segmentation branch network is used to extract first visual features at multiple pyramid scale levels of the target image.

3. The method for segmenting directional camouflage objects as described in claim 2, characterized in that, The extracted first visual features and the common representations of the directional camouflage objects are fused to obtain fused features, specifically: The first visual features at different pyramid scale levels are fused with the common representation through linear affine transformation; By using a long short-term memory network to fuse multiple cross-source information across scales, fused features are obtained.

4. The method for segmenting directional camouflage objects as described in claim 1, characterized in that, It also includes inputting the fused features and the reference mask into the reference feature enhancement module to enhance the fused features, specifically: Adjust the size of the reference mask and fused features to the pyramid scale of the first visual feature; The reference mask and fused features of the same scale are stitched together; By stitching together features of different scales, the enhanced fused features are obtained.

5. The method for segmenting directional camouflage objects as described in claim 4, characterized in that, The enhanced fused features are input into the decoder of the segmentation branch network for decoding, to obtain the segmentation result of the target image for the specified camouflaged object.

6. A directional camouflage object segmentation system, characterized in that, include: Reference branch network unit: used to extract features from multiple reference images of the directional camouflage object based on the reference branch network, to obtain the common characteristics of the directional camouflage object, specifically: The encoder of the reference branch network is used to encode multiple reference images to obtain the corresponding second visual features; The second visual feature is input into the decoder of the reference branch network to obtain the predicted foreground map of the directional camouflage object; The predicted foreground image is used to mask the second visual feature; Average pooling is performed on the masked second visual features to obtain the semantic vector representation of the directional camouflaged object. The common representation of directional camouflaged objects is obtained by averaging the semantic vector representations of directional camouflaged objects in multiple reference images. Segmentation branch network unit: used to input the target image including the directional camouflage object into the trained segmentation branch network for segmentation; the segmentation branch network includes an extraction unit, a fusion unit, a reference mask generation unit, and a segmentation unit; Extraction unit: used to extract first visual features from the target image including the directional camouflage object; The fusion unit is used to fuse the extracted first visual features and the common representations of the directional camouflage objects to obtain fused features; The reference mask generation unit is used to generate a reference mask based on the fusion feature and the common representation. Specifically, it uses dynamic convolution to take the fusion feature as input and the common representation as the convolution kernel, and compares each pixel of the common representation with the fusion feature to generate a reference mask. A segmentation unit is used to obtain a segmentation result of the target image based on the fusion features and the reference mask.

7. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform a directional camouflage object segmentation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs a directional camouflage object segmentation method as described in any one of claims 1 to 5.

Images