Knowledge-guided camouflage target detection methods and systems
By combining a multimodal large language model and a knowledge-guided semantic enhancement adapter module, the performance problem of camouflaged target detection under data scarcity conditions is solved, achieving high-quality camouflaged target segmentation and improved robustness.
Patent Information
- Application Number
- CN202510097090.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-22
- Publication Date
- 2025-11-14
- Estimated Expiration
- 2045-01-22
AI Technical Summary
Existing methods for detecting camouflaged targets are limited in performance under conditions of scarce data, and methods that rely on large-scale training data either ignore the problem of data scarcity or face the problem of noisy signals.
Multi-modal large language models are used to generate multi-level semantic knowledge descriptions. A knowledge-guided semantic enhancement adapter module is combined with a visual base model to enhance the semantic understanding ability of the camouflaged target detection model. The generality is maintained by freezing the parameters of the base model.
High-quality camouflaged target segmentation was achieved under low-data conditions, reducing dependence on training data and improving the robustness and accuracy of the model.
Smart Images
Figure CN120070855B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer and information service technology, and specifically relates to a method and system for perceiving and recognizing camouflaged targets in camouflaged images. Background Technology
[0002] Object detection is a fundamental task in computer vision, aiming to identify and locate objects in images. Camouflaged object detection (COD) is a fine-grained subtask within object detection, designed to detect objects that are difficult to detect hidden in the environment. Camouflaged scenarios are widespread in human production activities; for example, industrial products may have subtle surface defects, early lesions in medical images may be difficult to distinguish from normal tissue, and camouflaged pests in farmland may be difficult to spot with the naked eye. Therefore, camouflaged object detection has broad application prospects in practical production activities such as industrial defect detection, medical image segmentation, and agricultural pest and disease detection, and is thus attracting increasing attention from researchers in the field of computer vision.
[0003] Unlike general targets and salient targets, camouflaged targets usually have similar shapes, colors and textures to their surroundings. The high degree of intrinsic consistency between the target and the environment leads to sparse representation features for target perception. Therefore, the camouflaged target detection task is more difficult than general target detection and salient target detection, and requires more complex detection strategies.
[0004] Traditional camouflage target detection methods rely on low-level features such as brightness, intensity, color, texture, and optical flow. These methods have achieved reliable performance in specific scenarios, but they depend on manually designed operators, have limited feature extraction capabilities, and are difficult to handle complex backgrounds and significant target changes. Their accuracy and robustness are insufficient in different scenarios.
[0005] In recent years, deep learning-based methods for camouflaged target detection have made significant progress. Researchers have proposed various strategies to address the inherent consistency between the target and the background, which can be broadly categorized as: multi-scale feature mining-based methods, iterative methods, edge-guided methods, frequency domain information-based methods, uncertainty learning-based methods, and graph learning-based methods. Multi-scale feature mining-based methods extract contextual information from images at different scales, achieving representation enhancement by aggregating low-level detail features and high-level semantic features. Iterative methods, by simulating human visual mechanisms, divide camouflage detection into multiple stages such as discovery, localization, and refined perception, decomposing a difficult task into several simple steps to improve detection performance. Edge-guided methods enable the model to perceive and predict the contour edges of camouflaged objects, using edge supervision for constraints to achieve more accurate segmentation results. Frequency domain information-based methods, based on the idea that different frequency domain features of images can help discover hidden targets, utilize Fourier transforms to convert images from the spatial domain to the frequency domain, improving the model's detection capability by absorbing discriminative information from frequency domain features. Uncertainty learning-based methods model the uncertainty of model predictions, strengthening the loss penalty for low-confidence regions, causing the model to focus on confusing regions that are difficult to discriminate. Graph learning-based methods project the pixel space onto a graph space, and use the graph structure to uncover high-level relationships between pixels in a region.
[0006] Most deep learning-based methods focus on designing complex network structures and learning strategies to extract perceptual cues from camouflaged targets, relying on large-scale training datasets for training and achieving good performance results. However, on the one hand, camouflaged samples are far fewer than general samples; on the other hand, the inherent difficulty of camouflaged targets makes annotation more costly, and camouflaged samples in specific domains, such as medical images, require annotation by professionals. These factors lead to a shortage of high-quality labeled data in real-world applications, and the performance of complex structures specifically designed for camouflage scenarios is often limited under data-scarce conditions.
[0007] With the impressive power of base models trained on large-scale datasets, leveraging these models to solve the COD (Cost of Detection) problem has become a hot topic among researchers in the field. This mainly falls into two categories: directly applying base models to solve COD tasks and alleviating data scarcity. Some methods directly utilize the powerful representation and generalization capabilities of base models, fine-tuning them on camouflage datasets to achieve significant performance improvements. For example, CamoDiff treats camouflage target detection as a conditional generation task, utilizing the high generalization ability of diffusion models to handle complex, varied, and significantly different camouflage images, and further enhancing model generalization through uncertain supervision. Other methods aim to explore how to alleviate data scarcity using base models. For instance, LAKE-RED clusters target features and uses them as anchors for the background environment, using diffusion models to generate image backgrounds similar to the foreground targets, thus generating camouflage images from generic images to alleviate the data shortage problem. WSCOS uses the SAM model to obtain segmentation pseudo-labels to reduce the annotation cost of camouflage images and learns robust representations from noisy supervision. ProMaC leverages the rich intrinsic knowledge of multimodal large language models to mine semantic information of targets, proposing an iterative cue-mask generation framework to generate predictive masks without training. However, methods that directly utilize the base model to solve COD tasks ignore the problem of data scarcity, while methods that alleviate data scarcity through the base model face the problem of noisy signals. Summary of the Invention
[0008] To overcome the shortcomings of existing methods, this invention proposes a knowledge-guided camouflage target detection method to achieve high-quality segmentation of camouflage targets and reduce dependence on training data. Addressing the problem of sparse perceptual features for camouflage targets, this invention utilizes a multimodal large language model to generate knowledge descriptions for camouflage images. Through highly abstract and generalized knowledge representations, it strengthens the model's understanding of semantic targets and camouflage scenes, enhancing its semantic information representation capabilities. Specifically, a multi-level knowledge aggregation module generates multi-level semantic knowledge vectors by aggregating consistency information from multi-level text descriptions, effectively suppressing noise interference from erroneous descriptions and preventing overly rich text descriptions from causing the model's features to lose focus. To maintain the inherent knowledge and general capabilities of the base model and improve performance under low data resources, this invention proposes a knowledge-guided semantic enhancement adapter module. By freezing the original parameters of the base model and adjusting the image visual feature representation in the low-rank space, it guides the model's adjustment with semantic knowledge, integrating the semantic information of camouflage images while maintaining the general knowledge and segmentation capabilities of the visual model.
[0009] This invention uses image descriptions generated by a multimodal large language model as auxiliary information to guide the model to better understand camouflaged images, achieving high-quality camouflaged target segmentation. Simultaneously, an efficient parameter fine-tuning method based on domain adaptation preserves the intrinsic knowledge of the visual base model, effectively alleviating the dependence on large-scale training data.
[0010] The technical solution of this invention is a knowledge-guided method for detecting camouflaged targets, comprising the following steps:
[0011] Step 1, Dataset Construction: Select the target dataset to disguise the dataset, and divide the dataset into training set, validation set and test set;
[0012] Step 2: Use the defined task instructions to query the multimodal large language model and extract multi-level semantic knowledge descriptions;
[0013] Step 3: Construct a camouflaged target detection model. The specific process is as follows: The multi-level semantic knowledge description is encoded into a description vector by a text encoder. Then, the description vector is input into the multi-level knowledge aggregation module and aggregated into a multi-level knowledge vector. The multi-level knowledge vector is semantically guided by several knowledge-guided semantic enhancement adapter modules and parallel blocks of the encoding layer in the visual base model to obtain semantically enhanced image features. Finally, the prediction mask is decoded and output by the decoding layer in the visual base model.
[0014] Step 4: Input the training images and corresponding knowledge descriptions from the training set into the camouflage target detection model, perform supervised training using the annotations of the training images, test the performance on the validation set, and save the model that performs best on the validation set.
[0015] Step 5: Input the test images and corresponding knowledge descriptions from the test set into the best-performing model, and the model predicts the camouflaged target and outputs a segmentation mask.
[0016] Furthermore, the dataset includes datasets under normal settings and datasets under low data conditions. In the datasets under normal settings, the general method of setting up datasets in the field of camouflage target detection is followed. The camouflage images of the training sets in the COD10K and CAMO datasets are used as the training set, and the validation sets and test sets of the COD10K, CAMO and NC4K datasets are used as the validation set and test set, respectively.
[0017] In datasets with low data conditions, several classes are randomly selected from the COD10K dataset as base classes, and samples from other classes are used as new classes. For base class classes, multiple samples are randomly selected from the base class samples in the COD10K training set to form a small training set, and all base class samples in the COD10K test set are used as the base class test set. For new class classes, all new class samples in the COD10K test set are used to form the new class test set, and the base class test set is used as the validation set. Model training and validation are both performed on the base classes.
[0018] Furthermore, when using a multimodal large language model to query the text description of a camouflaged image, the task instructions consist of three parts: [task prompt], [query], and [format instruction]. Specifically, the [task prompt] prompts the multimodal large language model to activate its ability to process specific tasks by summarizing the content of a particular task; the [query] is used to ask for key information that humans need to perceive when identifying camouflaged objects, including category, quantity, color, texture, shape, and environment; and the [format instruction] is used to constrain the output response of the multimodal large language model so that its output has a fixed format.
[0019] Furthermore, for the obtained knowledge description, rich knowledge description information is obtained by querying the multimodal large language model through task instructions. The description information includes six aspects: category, quantity, color, texture, shape and environment. Then, the fixed prefix in the instruction is removed from the output text with a fixed template, and the text encoder of the CLIP model is used to encode it into a description vector. The description vector is then input into the multi-level knowledge aggregation module to be aggregated into a multi-level knowledge vector.
[0020] Furthermore, the specific processing procedure of the multi-level knowledge aggregation module is as follows: Taking D... a D b D c D d D e , Six embedded description vectors representing category, quantity, color, texture, shape, and environment are used. The category and quantity vectors are concatenated to obtain the complete category information of the entire image, which is then aggregated into a category knowledge vector through a multilayer perceptron. The category description information is multiplied by the color, texture, and shape descriptions to enhance consistency information and suppress interference from non-consistent noise information. After activation by the sigmoid activation function, they are integrated and finally aggregated into a target knowledge vector through a multilayer perceptron. The category description information is removed from the environment description information to eliminate the influence of target semantic information, and then integrated into an environment knowledge vector through a multilayer perceptron.
[0021] The aggregation process of category knowledge vectors and environment knowledge vectors is represented as follows:
[0022] K a=MLP(Cat(LN(D) a ), LN(D b )))
[0023] K c =MLP(LN(D f )-LN(D a ))
[0024] Where LN(·) and MLP(·) are linear layer and multilayer perceptron, respectively, and Cat(·) represents the splicing operation;
[0025] The aggregation process of the target knowledge vector is represented as follows:
[0026] Agg(D)=σ(N(D a )*LN(D))
[0027] K b =MLP(Agg(D) c )+Agg(D d )+Agg(D e ))
[0028] Where D represents any one of the color, texture, or shape description vectors, Agg(·) represents the consistency information enhancement process, and σ(·) is the sigmoid activation function.
[0029] 6. The knowledge-guided camouflage target detection method according to claim 4, characterized in that: the overall process of semantic guidance is expressed as:
[0030] x l =E l (x l-1 )+KSEA l (x l-1 )
[0031] in, This represents the output feature of the (l-1)th parallel block, which is obtained by adding the output features of the (l-1)th coding layer and the output features of the (l-1)th semantic enhancement adapter module. h, w, and c represent the length, width, and number of channels of the output feature, respectively. l (·) represents the coding layer in the l-th parallel block, KSEA l (·) represents the knowledge-guided semantic enhancement adapter module in the l-th parallel block; where the input of the first parallel block is the embedding features of the camouflaged image, which is obtained by performing patch embedding on the input camouflaged image, and the patch embedding is implemented through a convolutional layer.
[0032] Furthermore, the specific processing procedure of the knowledge-guided semantic enhancement adapter module KSEA is as follows:
[0033] For input features Projection is performed to emphasize semantic information at different levels. A 1x1 convolution maps the features to a low-rank space, and two 3x3 convolutions restore local details, enhancing feature representations at different semantic levels. This results in enhanced features x guided by category knowledge. a =Conv3(Conv3(Conv1(K) a *x))), Enhanced features x guided by target knowledge b =Conv3(Conv3(Conv1(K) b *x))), Enhanced features x guided by background knowledge c =Conv3(Conv3(Conv1(K) c *x))), where K a K b K c These represent the aggregated knowledge vectors at three levels: category, target, and environment; the enhanced feature x guided by category and target knowledge. a x b The features are added together and then subjected to two 3x3 convolutions to fuse global semantic features and target detail features, resulting in... Enhance feature x ab Features x guided by background knowledge c The stitching integrates the differences between the foreground and background, adjusts the channels using a 1x1 convolution, and enhances the distinction between the foreground and background using channel attention and spatial attention, as shown below:
[0034] x′=SA(CA(Conv1(Cat(x ab x c ))))
[0035] Where Conv3 represents a 3x3 convolution and Conv1 represents a 1x1 convolution. These are the output features of the knowledge-guided semantic enhancement adapter module. h, w, and c represent the length, width, and number of channels of the output feature, respectively. CA(·) and SA(·) represent the channel attention module and the spatial attention module, respectively.
[0036] Furthermore, cross-entropy loss L is adopted. BCE The loss L is the sum of the sums ... IOU The training process is performed, and the overall loss function is:
[0037] Loss = L BCE +L IOU
[0038] Cross-entropy loss and cross-union ratio loss are expressed as follows:
[0039] LBCE =-∑(P(x,y)*ln(G(x,y))+(1-P(x,y))*ln(1-G(x,y)))
[0040]
[0041] Where P(x, y) and G(x, y) represent the predicted mask value and the ground truth mask value at coordinates (x, y), respectively.
[0042] Furthermore, during training, the input samples are randomly cropped, randomly rotated, and randomly selected to achieve data augmentation. Specifically, random flipping is performed to horizontally flip the image with a certain probability, and random cropping is performed with a length and width not exceeding k pixels. Integers are randomly sampled from [Hk, H] as the length of the cropped image, and integers are randomly sampled from [Wk, W] as the width of the cropped image, where H and W represent the length and width of the original image, and the cropping is performed with the center of the original image as the center. Random rotation is performed with a certain probability, and integer angles are randomly sampled within the angle range to rotate the cropped image.
[0043] The enhanced image is input into the constructed KGDA camouflage target detection model to generate a camouflage target prediction mask. The loss is calculated through the loss function, and the parameters are updated by backpropagation. After several rounds of training, the model performance is tested on the validation set, and the model with the best performance on the validation set is saved, that is, the model with the smallest mean absolute error between all predicted masks and ground truth masks.
[0044] This invention also provides a knowledge-guided camouflage target detection system, comprising:
[0045] The processor and memory are used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the knowledge-guided camouflage target detection method as described in the above technical solution.
[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0047] a) A knowledge-guided camouflage target detection method is proposed, which utilizes a multimodal large language model to obtain multi-level semantic knowledge, helping the model understand camouflage images and segment targets.
[0048] (b) A multi-level knowledge aggregation module is proposed to integrate noisy text representations into multi-scale knowledge representation vectors, thereby mitigating the impact of ambiguity and inaccuracy in text descriptions and providing effective knowledge guidance.
[0049] c) A knowledge-guided semantic enhancement adapter module is proposed to align textual representations with visual representations and inject multi-level semantic knowledge into the visual model to enhance the model's semantic understanding.
[0050] d) By freezing the pre-trained parameters of the visual backbone and fine-tuning a small number of parameters, the general performance of the model can be maintained, achieving good performance under low data conditions. Attached Figure Description
[0051] Figure 1 This is the KGDA model structure of the present invention.
[0052] Figure 2 This invention proposes a multi-level knowledge aggregation module structure.
[0053] Figure 3 This invention presents a knowledge-guided semantic enhancement adapter module structure.
[0054] Figure 4 This is a visualization of the comparative experiments of this invention. The first column is the input sample, the second column is the ground truth mask, the third column is the prediction mask of the proposed model, and the fourth to tenth columns are the prediction masks of other models. Our method achieves more accurate prediction output. Detailed Implementation
[0055] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0056] like Figure 1 As shown, this embodiment of the invention provides a knowledge-guided method for detecting camouflaged targets, comprising the following steps:
[0057] Step 1, Dataset Construction: Select the target dataset to disguise the dataset, and divide the dataset into training set, validation set and test set;
[0058] Step 2, Description Text Extraction: Use the defined task instructions to query the multimodal large language model and extract multi-level semantic knowledge descriptions;
[0059] Step 3, Model Construction: Construct a KGDA camouflage target detection model. The specific process is as follows: Encode multi-level semantic knowledge descriptions into description vectors using a text encoder. Then, use the multi-level knowledge aggregation module MLKA to aggregate the description vectors into multi-level knowledge vectors. Use several knowledge-guided semantic enhancement adapter modules KSEA and parallel blocks of the encoding layer in the visual base model to semantically guide the multi-level knowledge vectors, obtaining semantically enhanced image features. Finally, decode the output prediction mask through the decoding layer in the visual base model. Step 4, Model Training: Input the training images and corresponding knowledge descriptions from the training set into the KGDA camouflage target detection model. Perform supervised training using the annotations of the training images. Test the performance on the validation set and save the model that performs best on the validation set.
[0060] Step 5, Model Testing: Input the test images and corresponding knowledge descriptions from the test set into the best-performing model, and the model predicts the camouflaged target and outputs a segmentation mask.
[0061] Furthermore, step 1 includes the following steps:
[0062] In the standard setup, following the general approach to dataset setup in the field of camouflage target detection, camouflage images from the training sets of the COD10K and CAMO datasets are used as the training set. The validation and test sets from the COD10K, CAMO, and NC4K datasets are used as the validation and test sets, respectively. For low-data testing, 36 classes are randomly selected from the COD10K dataset as base classes, and samples from other classes are used as new classes. For base class classes, 8 samples are randomly selected from the base class samples in the COD10K training set to form a small training set, and all base class samples from the COD10K test set are used as the base class test set. For new classes, all new class samples from the COD10K test set are used to form the new class test set. The validation set is similarly structured. In this low-data setup, both model training and validation are performed on the base class datasets.
[0063] Furthermore, step 2 includes the following:
[0064] Each image sample in the dataset is input into the multimodal large language model GPT-4o. A unified task instruction is used to query the large model, prompting it to provide a multi-level knowledge description of the input camouflaged image. The task instruction is designed based on the human cognitive path for describing camouflaged images and consists of three parts: [task prompt], [query], and [format instruction]. Specifically, the [task prompt] summarizes the content of a specific task to encourage the model to activate its ability to process that task. The [query] queries for key information that humans need to perceive when identifying camouflaged objects, including category, quantity, color, texture, and background environment. The [format instruction] constrains the model's output response, ensuring that the output contains a description with a fixed format. The specific instructions used in this invention are as follows:
[0065]
[0066] Furthermore, the KGDA camouflage target detection model in step 3 includes the following:
[0067] The KGDA camouflage target detection model includes a visual base model SAM (Segment Anything Model), a text encoder CLIP, a multi-level knowledge aggregation module, and a knowledge-guided semantic enhancement adapter.
[0068] For the multi-level knowledge aggregation module, in order to suppress noise introduced by potentially erroneous or illusory descriptions in the generated text descriptions, and to avoid excessively rich semantic knowledge diverting the model's focus, this module integrates the extracted rich text semantic descriptions into fixed multi-level knowledge vectors, including aggregated information at three levels: category, target, and environment. For example... Figure 1 As shown, we obtain rich knowledge description information from the multimodal large language model by querying the task instructions. This description information includes six aspects: category, quantity, color, texture, shape, and environment. Next, the fixed prefixes in the instructions are removed from the output text with a fixed template, and the text encoder of the CLIP model is used to encode it into a description vector. This rich text description information is input into the multi-level knowledge aggregation module to aggregate into a multi-level knowledge vector, thereby enhancing consistency information and suppressing noise interference. Specifically, taking D... a D b D c D d D e , These represent six embedded description vectors: category, quantity, color, texture, shape, and environment, where c represents the number of channels, and K... a K b , These represent the aggregated knowledge vectors at three levels: category, target, and environment, where c′ = c / 2. For example... Figure 2 As shown, the category and quantity vectors are concatenated to obtain the complete category information of the entire image, which is then aggregated into a category knowledge vector through a multilayer perceptron. The category description information is multiplied by color, texture, and shape descriptions respectively to enhance consistency information and suppress interference from inconsistent noise information. After activation by the sigmoid activation function, the data is integrated and finally aggregated into a target knowledge vector through a multilayer perceptron. Category description information is removed from the environmental description information to eliminate the influence of target semantic information, and the data is then integrated into an environmental knowledge vector through a multilayer perceptron.
[0069] The aggregation process of category knowledge vectors and environmental knowledge vectors is represented as follows:
[0070] K a =MLP(Cat(LN(D) a ), LN(D D )))
[0071] K c =MLP(LN(D f )-LN(D a ))
[0072] Where LN(·) and MLP(·) are linear and multilayer perceptrons, respectively, and Cat(·) represents the splicing operation.
[0073] The aggregation process of the target knowledge vector is represented as follows:
[0074] Agg(D)=σ(N(D a )*LN(D))
[0075] K b =MLP(Agg(D) c )+Agg(D d )+Agg(D e ))
[0076] Where D represents any one of the color, texture, or shape description vectors, Agg(·) represents the consistency information enhancement process, and σ(·) is the sigmoid activation function.
[0077] The above describes the knowledge description process. Next, we will introduce the image input processing process. For the input camouflage image... First, patch embedding is performed. Specifically, a convolutional layer with a kernel size of 16*16 and a stride of 16 is used to embed the image into the feature space to obtain the embedded features. Where H and W represent the length and width of the original image, and h and w represent the length and width of the embedded features, with h = H / 16 and w = W / 16.
[0078] For the knowledge-guided semantic enhancement adapter module, we insert it as a parallel module into the basic visual model. By freezing the core parameters of the model, we maintain the inherent knowledge and general capabilities of the basic model pre-trained on large-scale data. By training a small number of parameters of the semantic enhancement adapter module and the multi-level knowledge aggregation module, we achieve the model's adaptation to the spoofing detection task, realizing efficient fine-tuning under low data resources and accurate recognition under general settings. The overall process of semantic guidance is represented as follows:
[0079] x l =E l (x l-1 )+KSEA l (x l-1 )
[0080] in, This represents the output feature of the (l-1)th parallel block, which is obtained by adding the output features of the (l-1)th coding layer and the output features of the (l-1)th semantic enhancement adapter module. h, w, and c represent the length, width, and number of channels of the output feature, respectively. l (·) represents the coding layer in the l-th parallel block, KSEA l (·) represents the knowledge-guided semantic enhancement adapter module in the l-th parallel block; where the input of the first parallel block is the embedding features of the camouflaged image.
[0081] The knowledge-guided semantic enhancement adapter applies multi-layered semantic knowledge to guide the model towards consistency, enhancing its ability to understand camouflaged images by injecting semantic knowledge. Specifically, such as... Figure 3 As shown, we utilize the multi-level knowledge vectors aggregated by the multi-level knowledge aggregation module to aggregate the input features. Projection is performed to emphasize different levels of semantic information in the projected features. A 1x1 convolution maps the features to a low-rank space, and two 3x3 convolutions restore local details, enhancing feature representations at different semantic levels. Enhanced features x guided by category knowledge. a =Conv3(Conv3(Conv1(K) a *x))), Enhanced features x guided by target knowledge b =Conv3(Conv3(Conv1(K) b *x))), Enhanced features x guided by background knowledge c =Conv3(Conv3(Conv1(K) c *x))). Where x a x b , c′=c / 16. Enhanced features x guided by category and target knowledge. a x b The features are summed and then processed through two 3x3 convolutions to fuse global semantic features and target detail features. Enhance feature x ab Features x guided by background knowledge c The stitching integrates the differences between the foreground and background, adjusting the channels through a 1x1 convolution, and utilizing channel attention and spatial attention to enhance the distinction between the foreground and background. This is represented as:
[0082] x′=SA(CA(Conv1(Cat(x ab x c ))))
[0083] in These are the output features of the knowledge-guided semantic enhancement adapter, where CA(·) and SA(·) represent the channel attention module and the spatial attention module, respectively.
[0084] After several coding layers and a semantic enhancement adapter module, the final image coding features are obtained, which are then decoded by the SAM model's decoder to output a prediction mask. Cross-entropy loss L is employed. BCE The loss L is the sum of the sums ... IOU The training process is performed, and the overall loss function is:
[0085] Loss = L BCE +L IOU
[0086] Cross-entropy loss and cross-union ratio loss are expressed as follows:
[0087] L BCE =-∑(P(x,y)*ln(G(x,y))+(1-P(x,y))*ln(1-G(x,y)))
[0088]
[0089] Where P(x, y) and G(x, y) represent the predicted mask value and the ground truth mask value at coordinates (x, y), respectively.
[0090] Furthermore, model training in step 4 includes the following steps:
[0091] During training, to increase data diversity, we performed data augmentation steps such as random cropping and random rotation on the input samples. Specifically, random flipping was performed with a probability of 0.5, horizontally flipping the image. Random cropping was performed with a maximum of 30 pixels in length and width, using randomly sampled integers from [H-30, H] as the length of the cropped image and from [W-30, W] as the width of the cropped image, where H and W represent the original image length and width, respectively, with the original image center as the cropping center. Random rotation was performed with a probability of 0.2, rotating the cropped image with randomly sampled integers from -15° to 15°.
[0092] The enhanced image is input into the constructed KGDA model to generate a camouflaged target prediction mask. The loss is calculated using a loss function, and backpropagation is performed to update the parameters. After every 3 rounds of training, the model performance is tested on a validation set, and the model with the best performance on the validation set, i.e., the model with the smallest mean absolute error between all predicted masks and the ground truth mask, is saved.
[0093] This invention provides a knowledge-guided method for detecting camouflaged targets. It utilizes descriptive knowledge to help the model understand camouflaged images, aggregates rich image descriptions and suppresses noise interference through a multi-level knowledge aggregation module, and injects semantic guidance knowledge through a knowledge-guided semantic enhancement adapter module, ultimately achieving more accurate segmentation results. In addition, this method has more significant advantages under low data conditions.
[0094] The following specific example illustrates the implementation process and experimental results of this invention.
[0095] Step S1: Prepare the normal settings and low data settings datasets for model training.
[0096] a. Prepare three widely used camouflage target datasets: CAMO, COD10K, and NC4K. CAMO contains 1000 training images and 250 test images. COD10K contains 10000 images, of which 5066 are camouflage images, with 3040 training images and 2026 test images among the camouflage images. NC4K contains only the test set, with 4121 camouflage test samples.
[0097] b. Prepare the dataset under normal settings. Combine the training sets of COD10K and CAMO as the training set, with a total of 4040 training images. Use the COD10K test set as the validation set for the experiment. Use the test sets of COD10K, CAMO, and NC4K as the test sets for the experiment.
[0098] c. Prepare the dataset under low-data conditions. Randomly select 36 classes from the COD10K dataset as base classes, and use samples from other classes as new classes. For each base class, randomly select 8 samples from the base class samples in the COD10K training set to form a small training set, totaling 288 training images. Use all base class samples from the COD10K test set as the base class test set. For each new class, use all new class samples from the COD10K test set to form the new class test set. Use the base class test set as the validation set. Under low-data conditions, model training and validation are performed on the base class dataset.
[0099] The purpose of setting up a dataset with low data conditions is to obtain a dataset with low data conditions, that is, far less training data than in the normal setting, in order to verify the model's generalization ability with a small number of training samples.
[0100] Step S2: Input the dataset samples into the multimodal large language model and query the knowledge description of the camouflaged image as the text content for subsequent semantic guidance.
[0101] a. Our multimodal large language model uses GPT-4o, and we prepare image samples and task instructions;
[0102] b. By calling the application programming interface (API) provided by OpenAI, the GPT-4o model is accessed to query the knowledge description of each camouflaged image, and the generated text is saved in a txt file.
[0103] Step S3: Set up a text encoder to encode the text description, and set up a visual base model to encode image features and decode the output prediction results.
[0104] a. The text encoder adopts the CLIP model text encoder to encode knowledge descriptions into representation vectors;
[0105] b. The visual base model adopts the SAM model, which is pre-trained on more than 11 million images and has rich visual knowledge and powerful segmentation capabilities, used to predict the segmentation mask of camouflaged targets.
[0106] c. Freeze the decoder parameters of SAM, and do not update the parameters during training. The decoder parameters will be trained normally.
[0107] d, The final prediction mask generated by the model is constrained by a joint loss of cross-entropy loss and cross-union ratio loss to ensure that the model prediction results gradually approach the true value during training.
[0108] Step S4: Construct a multi-level knowledge aggregation module. Integrate noisy text representations into multi-scale knowledge representation vectors. By aggregating consistent information, suppress the influence of irrelevant or erroneous information, alleviate interference caused by ambiguity and inaccuracies in text descriptions, and provide effective knowledge guidance.
[0109] a. Obtain the multi-dimensional description vector. Input the text description obtained in step S2 into the CLIP token processor to map the token. The token is then encoded into a description vector by the CLIP text encoder.
[0110] b) Aggregate description vectors into multi-level knowledge vectors. Construct linear and multi-layer perceptron layers. Category descriptions and quantity descriptions are aggregated into category knowledge; category descriptions, color descriptions, texture descriptions, and shape descriptions are aggregated into target knowledge; and category descriptions and environment descriptions are aggregated into environment knowledge. Rich text descriptions are aggregated into compact knowledge vectors, and consistency information is used to reduce noise interference and provide effective knowledge guidance.
[0111] Step S5: Construct a knowledge-guided semantic enhancement adapter module to align textual representations with visual representations, integrate multi-level semantic knowledge into the visual model, guide the model to adjust the feature space using semantic knowledge, and help the model understand camouflaged images through semantic injection, thereby better adapting to downstream tasks of camouflaged target detection.
[0112] a) In each encoding layer of the SAM model encoder, a side-branch knowledge-guided semantic enhancement adapter module is constructed to adjust the image feature space of each layer.
[0113] b) Multiply the category knowledge vector, target knowledge vector, and environment knowledge vector with image features respectively to enhance semantic information at different levels.
[0114] c. Construct a 1*1 convolutional layer and two 3*3 convolutional layers to map the semantic enhancement features to a low-rank space, and perform feature enhancement processing in the low-rank space.
[0115] d. Category enhancement features and target enhancement features are fused together and then concatenated with environment enhancement features to enrich the semantic information of the disguised target and distinguish the differences between the target and the environment.
[0116] e. Construct a 1*1 convolutional layer, a channel attention layer, and a spatial attention layer to enhance the difference features between the target and the background and improve the model's camouflage perception ability.
[0117] Model training was implemented on the Ubuntu operating system, using the PyTorch deep learning framework to build the model and utilizing the GPU for computation. The specific hardware and software parameters used in this example are shown in the table below. This example uses the Adam optimizer, with a batch size of 6, a learning rate of 0.0002, and performs optimization for 60 epochs.
[0118]
[0119] During model training, the training, validation, and test sets were divided according to step 1, constructing both a standard dataset and a low-data-set dataset. Experiments were conducted on both datasets. Data augmentation was performed on the input images. The augmented images were then uniformly resized to 704*704 and input into the constructed model along with their corresponding descriptive text.
[0120] (1) After the image is input into the model, a 1*704*704 prediction mask is finally obtained. The cross-entropy loss and cross-union ratio loss between the prediction mask and the ground truth mask are calculated, and the two losses are added together as the overall loss:
[0121] Loss = L BCE +L IOU
[0122] L BCE =-∑(P(x,y)*ln(G(x,y))+(1-P(x,y))*ln(1-G(x,y)))
[0123]
[0124] Where P(x, y) and G(x, y) represent the prediction mask and the ground truth mask, respectively.
[0125] During training, after every three rounds of training, the model is validated on the validation set. The mean of the average absolute error of the model on all samples on the validation set is calculated, and the current test result is compared with the previous test result. If the current result is better than the previous result, the model is saved.
[0126] After the model is trained, its performance is tested on a test set. During testing, the camouflaged image samples are scaled to 704*704 pixels without data augmentation. Subjective and objective evaluations are used to assess the model's performance.
[0127] Objective evaluation is achieved through index calculation. The S-measure (Spredict) is calculated separately for the predicted mask and the ground truth mask. α F-measure (F) β Mean Absolute Error (MAE) and E-measure Used to evaluate the segmentation accuracy of the model.
[0128] S measures the spatial structural similarity between the prediction mask and the truth mask, which incorporates object-aware evaluation S. o and regional perception assessment S r , means as follows:
[0129] S α =α*S o +(1-α)*S r
[0130] Where α∈[0,1] is the weighting factor, which is usually set to 0.5.
[0131] The F-metric is used to calculate the relationship between precision and recall. It adjusts the mask to the range [0, 255] and divides it into a binary mask by a threshold, as shown below:
[0132]
[0133] Where M(T) represents binarizing the prediction mask using a threshold T. |·| represents the total area of the mask.
[0134] The F-metric is expressed as:
[0135]
[0136] β 2 It is usually set to 0.3.
[0137] The E-metric evaluates the local and global similarity between the predicted and ground truth masks, and is expressed as:
[0138]
[0139] in For the enhanced alignment matrix, W and H are the width and height of the input image. C and G are the prediction mask and ground truth mask, respectively.
[0140] The mean absolute error (MAE) measures the pixel-wise mean absolute error between the normalized predicted mask and the ground truth mask, expressed as:
[0141]
[0142] Where W and H are the height and width of the input image, and |·| represents the absolute value operation. C and G are the prediction mask and the ground truth mask, respectively.
[0143] By calculating various indicators and comparing their magnitudes with those of other models, the detection performance of the model is reflected. The method of this invention achieves the optimal evaluation index among 15 comparative models.
[0144] Subjective evaluation compares the visual effects of predictive masks, from Figure 4 As can be seen, the method of the present invention achieves more accurate segmentation results and has significant advantages in detecting small targets, confusing targets, and multiple targets.
[0145] On the other hand, embodiments of the present invention also provide a knowledge-guided camouflage target detection system, including:
[0146] The processor and memory are used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the knowledge-guided camouflage target detection method as described in the above technical solution.
[0147] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.
Claims
1. A knowledge-guided method for detecting camouflaged targets, characterized in that, Includes the following steps: Step 1, Dataset Construction: Select the target dataset to disguise the dataset, and divide the dataset into training set, validation set and test set; Step 2: Use the defined task instructions to query the multimodal large language model and extract multi-level semantic knowledge descriptions; Step 3: Construct a camouflaged target detection model. The specific process is as follows: The multi-level semantic knowledge description is encoded into a description vector by a text encoder. Then, the description vector is input into the multi-level knowledge aggregation module and aggregated into a multi-level knowledge vector. The multi-level knowledge vector is semantically guided by several knowledge-guided semantic enhancement adapter modules and parallel blocks of the encoding layer in the visual base model to obtain semantically enhanced image features. Finally, the prediction mask is decoded and output by the decoding layer in the visual base model. The specific processing procedure of the multi-level knowledge aggregation module is as follows: , , , , , Six embedded description vectors representing category, quantity, color, texture, shape, and environment are used. The category and quantity vectors are concatenated to obtain the complete category information of the entire image, which is then aggregated into a category knowledge vector through a multilayer perceptron. The category description information is multiplied by the color, texture, and shape descriptions to enhance consistency information and suppress interference from non-consistent noise information. After activation by the sigmoid activation function, they are integrated and finally aggregated into a target knowledge vector through a multilayer perceptron. The category description information is removed from the environment description information to eliminate the influence of target semantic information, and then integrated into an environment knowledge vector through a multilayer perceptron. The specific processing procedure of the knowledge-guided semantic enhancement adapter module is as follows: For input features Projection is performed so that the projected features emphasize semantic information at different levels. After 1...
1. Convolution maps features to a low-rank space, through two 3.
3. Convolution restores local details and enhances feature representations at different semantic levels, thereby obtaining enhanced features guided by category knowledge. Enhanced features guided by target knowledge Enhanced features guided by background knowledge ,in , , These represent the aggregated knowledge vectors at three levels: category, target, and environment; and the enhanced features guided by category and target knowledge. , Add and pass through two 3s 3. Convolution is used to fuse global semantic features and target detail features to obtain... ; will enhance features Features guided by background knowledge The splicing integrates the differences between the foreground and background, after 1 1. Convolution adjusts channels, utilizing channel attention and spatial attention to enhance the distinction between foreground and background; Step 4: Input the training images and corresponding knowledge descriptions from the training set into the camouflage target detection model, perform supervised training using the annotations of the training images, test the performance on the validation set, and save the model that performs best on the validation set. Step 5: Input the test images and corresponding knowledge descriptions from the test set into the best-performing model, and the model predicts the camouflaged target and outputs a segmentation mask.
2. The knowledge-guided camouflage target detection method according to claim 1, characterized in that: The dataset includes datasets under normal settings and datasets under low data conditions. In the datasets under normal settings, the general method of setting up datasets in the field of camouflage target detection is followed. The camouflage images of the training sets in the COD10K and CAMO datasets are used as the training set, and the validation sets and test sets of the COD10K, CAMO and NC4K datasets are used as the validation set and test set, respectively. In datasets with low data conditions, several classes are randomly selected from the COD10K dataset as base classes, and samples from other classes are used as new classes. For base class classes, multiple samples are randomly selected from the base class samples in the COD10K training set to form a small training set, and all base class samples in the COD10K test set are used as the base class test set. For new class classes, all new class samples in the COD10K test set are used to form the new class test set, and the base class test set is used as the validation set. Model training and validation are both performed on the base classes.
3. The knowledge-guided camouflage target detection method according to claim 1, characterized in that: When using a multimodal large language model to query the text description of a camouflaged image, the task instructions consist of three parts: [task prompt], [query], and [format instructions]. Specifically, the [task prompt] prompts the multimodal large language model to activate its ability to process specific tasks by summarizing the content of the specific task; the [query] is used to ask for key information that humans need to perceive when identifying camouflaged objects, including category, quantity, color, texture, shape, and environment; and the [format instructions] are used to constrain the output response of the multimodal large language model so that its output has a fixed format.
4. The knowledge-guided camouflage target detection method according to claim 1, characterized in that: For the obtained knowledge description, rich knowledge description information is obtained by querying the multimodal large language model through task instructions. The description information includes six aspects: category, quantity, color, texture, shape and environment. Then, the fixed prefix in the instruction is removed from the output text with a fixed template, and the text encoder of the CLIP model is used to encode it into a description vector. The description vector is then input into the multi-level knowledge aggregation module to be aggregated into a multi-level knowledge vector.
5. The knowledge-guided camouflage target detection method according to claim 1, characterized in that: The aggregation process of category knowledge vectors and environmental knowledge vectors is represented as follows: in , These are linear layer and multilayer perceptron, respectively. Indicates a splicing operation; The aggregation process of the target knowledge vector is represented as follows: in, It represents any one of the vectors describing color, texture, or shape. This represents the process of enhancing consistency information. It is the sigmoid activation function.
6. The knowledge-guided camouflage target detection method according to claim 1, characterized in that: The overall process of semantic guidance can be represented as follows: in, Indicates the first The output characteristics of the nth parallel block, which are determined by the nth parallel block. The output features of the layer coding layer and the first The output features of each semantic enhancement adapter module are added together to obtain the result. h , w、c These represent the length, width, and number of channels of the output feature, respectively. It is the first The coding layer in a parallel block For the first The knowledge-guided semantic enhancement adapter module is in several parallel blocks; the input of the first parallel block is the embedding features of the camouflaged image, which is obtained by patch embedding on the input camouflaged image, and the patch embedding is implemented through a convolutional layer.
7. The knowledge-guided camouflage target detection method according to claim 1, characterized in that: The specific processing procedure of the knowledge-guided semantic enhancement adapter module can be expressed by the following formula: in, Indicates 3 3 convolutions, Indicates 1 1 convolution, These are the output features of the knowledge-guided semantic enhancement adapter module. h , w、c These represent the length, width, and number of channels of the output feature, respectively. , These represent the channel attention module and the spatial attention module, respectively.
8. The knowledge-guided camouflage target detection method according to claim 1, characterized in that: Using cross-entropy loss And the loss of the intersection The training process is performed, and the overall loss function is: Cross-entropy loss and cross-union ratio loss are expressed as follows: in and Representing coordinates ( x , y The predicted mask value and the true mask value of the position.
9. The knowledge-guided camouflage target detection method according to claim 1, characterized in that: During training, input samples are randomly cropped, rotated, and selected to achieve data augmentation. Specifically, random flipping involves horizontally flipping the image with a certain probability, and random cropping is performed within k pixels in length and width. Integers are randomly sampled from [Hk, H] as the length of the cropped image and from [Wk, W] as the width of the cropped image, where H and W represent the length and width of the original image, and the cropping is centered on the original image. Random rotation involves rotating the cropped image with a certain probability, and random integer angles are sampled within the angle range to rotate the cropped image. The enhanced image is input into the constructed camouflage target detection model to generate a camouflage target prediction mask. The loss is calculated using a loss function, and backpropagation is performed to update the parameters. After several rounds of training, the model performance is tested on the validation set, and the model that performs best on the validation set, i.e., the model with the smallest mean absolute error between all predicted masks and ground truth masks, is saved.
10. A knowledge-guided camouflage target detection system, characterized in that, include: The processor and memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the knowledge-guided camouflage target detection method as described in any one of claims 1-9.