A multi-modal camouflage target detection method based on modal-independent prompt learning

Abstract

The present application relates to the technical field of computer vision, and is a multi-modal camouflage target detection method based on modal-independent prompt learning, comprising: fusing an RGB image and auxiliary modal features to generate content domain features; constructing a learnable prompt prototype and modulating the content domain features to generate prompt domain features; performing bidirectional interaction on the content domain features and the prompt domain features through a cross-attention mechanism; inputting the bidirectional interaction result into a segmentation model decoder to generate an initial target mask; generating a transformation coefficient using the prompt domain features after bidirectional interaction; combining the transformation coefficient to correct the initial target mask to generate a target mask, thereby realizing camouflage target detection. The present application can realize modal-independent unified modeling capability and effectively solve the problem of camouflage object detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multimodal camouflage target detection method based on modality-independent cue learning. Background Technology

[0002] The task of camouflaged target detection aims to segment targets that blend seamlessly into the environment from complex backgrounds. Traditional RGB methods struggle to extract effective foreground information in highly similar scenes. Existing multimodal methods enhance detection performance by incorporating depth, thermal infrared, or polarization information, but most rely on specific modal designs, lacking unified modeling capabilities and failing to adapt to different modal combinations. Furthermore, while SAM-based models possess good generalization capabilities, existing methods typically employ modality-specific adaptation structures, limiting cross-modal scalability. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a multimodal camouflage target detection method based on modality-independent cue learning. The invention constructs a content domain to fuse RGB and auxiliary modal features, and a cue domain to generate a learnable cue prototype. A dual-domain interaction module utilizes cross-attention to achieve synergistic enhancement of the content and cue domains. The enhanced features are input into the segmentation model decoder to generate an initial prediction. A mask refinement module, combined with cue information, corrects the prediction results. This method achieves unified modeling of any auxiliary modality, improving the generalization ability and accuracy of camouflage target detection.

[0004] The technical means employed in this invention are as follows: A multimodal camouflage target detection method based on modality-independent cue learning includes: fusing RGB images with auxiliary modal features to generate content domain features; constructing a learnable cue prototype and modulating it according to the content domain features to generate cue domain features; performing bidirectional interaction between the content domain features and the cue domain features through a cross-attention mechanism; inputting the bidirectional interaction result into a segmentation model decoder to generate an initial target mask; generating transformation coefficients using the cue domain features after bidirectional interaction; and correcting the initial target mask using the transformation coefficients to generate a target mask, thereby achieving camouflage target detection.

[0005] Furthermore, the generation of the content domain features specifically includes: extracting RGB images through a pre-trained SAM image encoder. The RGB features are extracted, and auxiliary modal features are extracted through a multi-layer feature extraction network:

[0006] in, The auxiliary modality image is represented by ; PVT represents the Transformer-based backbone network. This represents the four-layer features extracted through the PVT backbone network; the RGB features and auxiliary modal features are summed element-wise to form the content domain features.

[0007] Further, the learnable cue prototype is initialized. Modulation weights are generated based on content domain features through convolution and nonlinear mapping:

[0008] in, The sigmoid function is represented by CLG, convolution, normalization, and a continuous block of GeLU activation function, and A represents adaptive pooling. Representing content domain features, MLP stands for Multilayer Perceptron. The obtained modulation weights are represented; the learningable cue prototype is then applied using the modulation weights. Weighting is performed to form the feature of the prompt domain:

[0009] in, This indicates the characteristics of the prompt field.

[0010] Furthermore, the cross-attention mechanism calculates the bidirectional interaction results from the content domain to the cue domain and from the cue domain to the content domain, respectively; using the cue domain features as the query vector and the content domain features as the key-value vector, the mapping result from the content domain to the cue domain is... for:

[0011] in, This represents the Sigmoid function. This indicates that the suggestion field will be used as the query vector. This indicates that the content field is used as a key-value vector. This represents the dimension of the key-value vector. Represents the content domain; uses content domain features as the query vector, hint domain features as the key-value vector, and the mapping result from the hint domain to the content domain. for:

[0012] in, This represents the Sigmoid function. This indicates that the content field is used as the query vector. This indicates that the suggestion domain will be used as a key-value vector. This represents the dimension of the key-value vector. This indicates the prompt field.

[0013] Furthermore, the generation of the modulation coefficients specifically includes: inputting the bidirectional interaction result into the segmentation model decoder to generate an initial target mask; introducing an auxiliary token to enhance multimodal representation capabilities; inputting the auxiliary token and the original prompt information together into the segmentation model; and performing nonlinear mapping on the prompt domain features after bidirectional interaction to generate transform coefficients.

[0014] in, The cue domain features represent the characteristics after two-way interaction; MLP stands for Multilayer Perceptron. This represents the Sigmoid function. Represents the transformation coefficients.

[0015] Furthermore, the target mask is represented as:

[0016] in, Indicates the target mask. The transform coefficients are represented by CLG, which stands for convolution and is used for normalization. This represents the transposed convolutional features after SAM decoding. The SAM represents the features after the attention mechanism in the SAM decoder, and MLP represents the multilayer perceptron.

[0017] Compared with the prior art, the present invention has the following advantages: This invention constructs a content domain to fuse RGB and auxiliary modal features, and a cue domain to generate a learnable cue prototype. A dual-domain interaction module utilizes cross-attention to achieve synergistic enhancement of the content and cue domains. The enhanced features are input into a segmentation model decoder to generate an initial prediction. A mask refinement module, combined with cue information, corrects the prediction results. This method achieves unified modeling for any auxiliary modality, improving the generalization ability and accuracy of camouflaged target detection. This invention enables modality-independent unified modeling, effectively solving the problem of camouflaged object detection.

[0018] This invention effectively integrates data-driven information and prior knowledge through a bidirectional interaction mechanism between the content domain (data-driven) and the prompting domain (knowledge-driven); it also improves the accuracy of camouflaged target boundary detection by correcting the initial mask through a mask refinement module. Furthermore, this invention provides a scalable, multi-task unified framework, applicable not only to camouflaged target detection tasks but also extending to other segmentation tasks. Attached Figure Description To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the multimodal camouflage target detection method based on modality-independent cue learning in this invention.

[0020] Figure 2 This is an architecture diagram of the multimodal camouflage target detection method based on modality-independent cue learning in this invention. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0024] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0025] like Figure 1 As shown, this invention provides a multimodal camouflage target detection method based on modality-independent cue learning, comprising: fusing RGB images with auxiliary modality features to generate content domain features; specifically, in a preferred embodiment of this invention, the generation of the content domain features specifically includes: extracting RGB images through a pre-trained SAM image encoder. The RGB features are extracted, and auxiliary modal features are extracted through a multi-layer feature extraction network:

[0026] in, The auxiliary modality image is represented by ; PVT represents the Transformer-based backbone network. This represents the four layers of features extracted through the PVT backbone network; the RGB features and auxiliary modal features are summed element-wise to form the content domain features. The content domain integrates the RGB and auxiliary modal features to form a unified semantic representation.

[0027] A learnable prompt prototype is constructed and modulated according to the content domain features to generate prompt domain features; in a preferred embodiment of the present invention, the learnable prompt prototype is initialized. Modulation weights are generated based on content domain features through convolution and nonlinear mapping:

[0028] in, The sigmoid function is represented by CLG, convolution, normalization, and a continuous block of GeLU activation function, and A represents adaptive pooling. Representing content domain features, MLP stands for Multilayer Perceptron. This represents the obtained modulation weights; The learnable cue prototype is applied using the modulation weights. Weighting is performed to form the feature of the prompt domain:

[0029] in, Represent the cue domain features. Generate the cue domain and extract prior task information through learnable cue prototypes.

[0030] A cross-attention mechanism is used to enable bidirectional interaction between the content domain features and the cue domain features; for example... Figure 2 As shown, cross-attention is used to achieve the fusion of data-driven information and knowledge-driven information; in a preferred embodiment of the present invention, the cross-attention mechanism calculates the bidirectional interaction results from the content domain to the prompt domain and from the prompt domain to the content domain. Using the cue domain features as the query vector and the content domain features as the key-value vector, the mapping result from the content domain to the cue domain is... for:

[0031] in, This represents the Sigmoid function. This indicates that the suggestion field will be used as the query vector. This indicates that the content field is used as a key-value vector. This represents the dimension of the key-value vector. Represents the content field; Using content domain features as the query vector and suggestion domain features as the key-value vector, the mapping result from the suggestion domain to the content domain is shown. for:

[0032] in, This represents the Sigmoid function. This indicates that the content field is used as the query vector. This indicates that the suggestion domain will be used as a key-value vector. This represents the dimension of the key-value vector. This indicates the prompt field.

[0033] The bidirectional interaction result is input into the segmentation model decoder to generate an initial target mask; transform coefficients are generated using the cue domain features after bidirectional interaction; and the prediction result is corrected using cue information to improve boundary accuracy. Specifically, in a preferred embodiment of the invention, the generation of the modulation coefficients specifically includes: inputting the bidirectional interaction result into the segmentation model decoder to generate an initial target mask; introducing an auxiliary token to enhance multimodal representation capabilities; inputting the auxiliary token and the original cue information together into the segmentation model; and performing a nonlinear mapping on the cue domain features after bidirectional interaction to generate transform coefficients.

[0034] in, The cue domain features represent the characteristics after two-way interaction; MLP stands for Multilayer Perceptron. This represents the Sigmoid function. Represents the transformation coefficients.

[0035] The initial target mask is corrected using the transformation coefficients to generate a new target mask, thereby achieving camouflaged target detection. In a preferred embodiment of this invention, the target mask is represented as:

[0036] in, Indicates the target mask. The transform coefficients are represented by CLG, which stands for convolution and is used for normalization. This represents the transposed convolutional features after SAM decoding. The SAM represents the features after the attention mechanism in the SAM decoder, and MLP represents the multilayer perceptron.

[0037] Example 1 like Figure 1 As shown, the specific implementation steps of the method of the present invention include: S1. Content domain construction: Input RGB image and auxiliary modal data, extract multimodal features, and generate content domain features by fusing elements one by one.

[0038] S2. Cue Domain Generation: Initialize the learnable cue prototype and generate modulation weights based on the content domain features to obtain the cue domain features.

[0039] S3, Dual-domain interaction: The content domain and the prompt domain serve as query and key-value pairs for each other, and feature prompts are enhanced through a cross-attention mechanism.

[0040] S4. Segmentation Prediction: After inputting the enhanced features into the segmentation model SAM to obtain the initial segmentation mask, the mask is corrected using the cue domain features, and the final detection result is output.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal camouflaged target detection method based on modality-independent cue learning, characterized in that, include: RGB images are fused with auxiliary modal features to generate content domain features; A learnable prompt prototype is constructed and modulated according to the content domain features to generate prompt domain features; The content domain features and the cue domain features interact bidirectionally through a cross-attention mechanism; The bidirectional interaction results are input into the segmentation model decoder to generate the initial target mask; Transformation coefficients are generated using the features of the prompt domain after two-way interaction; The initial target mask is corrected by combining the transformation coefficients to generate a target mask, thereby achieving camouflaged target detection.

2. The multimodal camouflage target detection method based on modality-independent cueing learning according to claim 1, characterized in that, The generation of the content domain features specifically includes: RGB images are extracted using a pre-trained SAM image encoder. The RGB features are extracted, and auxiliary modal features are extracted through a multi-layer feature extraction network: in, The auxiliary modality image is represented by ; PVT represents the Transformer-based backbone network. This represents the four layers of features extracted through the PVT backbone network; The RGB features and auxiliary modal features are summed element-wise to form the content domain features.

3. The multimodal camouflage target detection method based on modality-independent cueing learning according to claim 1, characterized in that, Initialize the learnable prompt prototype Modulation weights are generated based on content domain features through convolution and nonlinear mapping: in, The sigmoid function is represented by CLG, convolution, normalization, and a continuous block of GeLU activation function, and A represents adaptive pooling. Representing content domain features, MLP stands for Multilayer Perceptron. This represents the obtained modulation weights; The learnable cue prototype is applied using the modulation weights. Weighting is performed to form the feature of the prompt domain: in, This indicates the characteristics of the prompt field.

4. The multimodal camouflage target detection method based on modality-independent cueing learning according to claim 1, characterized in that, The cross-attention mechanism calculates the bidirectional interaction results from the content domain to the cue domain and from the cue domain to the content domain, respectively. Using the cue domain features as the query vector and the content domain features as the key-value vector, the mapping result from the content domain to the cue domain is... for: in, This represents the Sigmoid function. This indicates that the suggestion field will be used as the query vector. This indicates that the content field is used as a key-value vector. This represents the dimension of the key-value vector. Represents the content field; Using content domain features as the query vector and suggestion domain features as the key-value vector, the mapping result from the suggestion domain to the content domain is shown. for: in, This represents the Sigmoid function. This indicates that the content field is used as the query vector. This indicates that the suggestion domain will be used as a key-value vector. This represents the dimension of the key-value vector. This indicates the prompt field.

5. The multimodal camouflage target detection method based on modality-independent cueing learning according to claim 1, characterized in that, The generation of the modulation coefficients specifically includes: The bidirectional interaction result is input into the segmentation model decoder to generate an initial target mask. An auxiliary token is introduced to enhance multimodal representation capabilities. The auxiliary token and the original cue information are input together into the segmentation model, and the cue domain features after bidirectional interaction are nonlinearly mapped to generate transformation coefficients. in, The cue domain features represent the characteristics after two-way interaction; MLP stands for Multilayer Perceptron. This represents the Sigmoid function. Represents the transformation coefficients.

6. The multimodal camouflage target detection method based on modality-independent cueing learning according to claim 1, characterized in that, The target mask is represented as: in, Indicates the target mask. The transform coefficients are represented by CLG, which stands for convolution and is used for normalization. This represents the transposed convolutional features after SAM decoding. The SAM represents the features after the attention mechanism in the SAM decoder, and MLP represents the multilayer perceptron.

Images