Dual-granularity semantic cueing infrared dim small target detection method

Abstract

The application provides a dual-granularity semantic prompt infrared dim small target detection method. A target detection network based on dual-granularity semantic prompt is constructed. First, an image encoder is used to extract multi-scale visual features of an input infrared image. Then, a dual-granularity semantic prompt mechanism is used, a general semantic context is provided by using a predefined template as a coarse-grained text prior, and a fine-grained semantic description related to image content is dynamically generated through an inversion network of a visual-text mapping strategy to form an image-adaptive semantic prompt. Next, a text-guided image decoder is used to fuse text and visual features through a cross-attention mechanism and a text-guided spatial attention mechanism. Finally, a binary segmentation mask is output to complete target detection. The application can accurately detect infrared dim small targets with image-related semantic prompts in a complex background without manual annotation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and infrared image processing technology, specifically relating to a dual-granularity semantic prompting method for detecting weak infrared targets. It is suitable for application scenarios with high requirements for accuracy and robustness in detecting weak infrared targets, such as military early warning, civilian monitoring, and disaster relief. Background Technology

[0002] Infrared Small Target Detection (IRSTD) aims to accurately identify and locate targets with low signal-to-noise ratio, small size, and lack of texture features from infrared images. Because infrared images are often affected by complex backgrounds, atmospheric noise, and thermal diffusion effects, traditional methods such as filtering-based algorithms (e.g., Top-Hat, LoG) or local contrast-based algorithms often perform poorly in complex scenes.

[0003] In recent years, deep learning-based methods have improved detection performance through multi-scale fusion and attention mechanisms, but they still heavily rely on visual features and lack high-level semantic understanding capabilities, resulting in insufficient generalization ability when the target is blurry or the background is complex. The paper "Huang F, Zheng S, Qiu Z, et al. Text-IRSTD: Leveraging SemanticText to Promote Infrared Small Target Detection in Complex Scenes[J]. 2025." attempts to introduce a vision-language model (such as CLIP) to provide semantic priors through text descriptions, but it still suffers from problems such as high annotation costs, biases in semantic descriptions, and inference dependence on annotations.

[0004] Therefore, existing technologies lack a method for small target detection that can achieve high robustness and high generalization ability in complex backgrounds, low signal-to-noise ratio, and target-blurred scenes using only the information of infrared images themselves, without the need for additional text annotation or high-level semantic priors. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a method for detecting small infrared targets with dual-granularity semantic cues. First, an image encoder extracts multi-scale visual features from the input infrared image. Then, a dual-granularity semantic cues mechanism is employed, using a predefined text template to provide a general semantic context as a coarse-grained text prior. A fine-grained semantic description related to the image content is dynamically generated through a visual-text mapping strategy inversion network, forming an image-adaptive semantic cues. Next, a text-guided image decoder is used, fusing text and visual features through a cross-attention mechanism and a text-guided spatial attention module. Finally, a binary segmentation mask is output to complete the detection of small infrared targets. This invention eliminates the need for manual text annotation and can accurately detect small infrared targets with image-related semantic cues in complex backgrounds.

[0006] A dual-granularity semantic cue infrared weak target detection method, characterized by the following steps: S1: Preprocess the input single infrared image to obtain a standardized infrared image; S2: Construct a target detection network based on dual-granularity semantic prompts, mainly including an image encoder, an image-text inversion network, and a text-guided image decoder; The image encoder performs multi-level feature extraction on the input image to obtain multi-scale visual features; The image-text inversion network and text encoder constitute a dual-granularity semantic prompt module, generating dual-granularity semantic prompts corresponding to the input image. Specifically, a text encoder with fixed template text and frozen parameters is used to obtain coarse-grained semantic features, and an image-text inversion network is used to map visual features to dynamically generate fine-grained semantic features. The coarse-grained semantic features and fine-grained semantic features constitute dual-granularity semantic prompts. The text-guided image decoder progressively fuses multi-scale visual features with dual-granularity semantic cues and outputs a segmentation mask for infrared weak targets. The text-guided image decoder is a five-level cascaded structure. Before the first level, a cross-attention module fuses high-level visual features and dual-granularity semantic cues. Each of the remaining levels includes upsampling and a text-guided spatial attention TGSA module. The specific processing is as follows: each level of the decoder first performs bilinear interpolation upsampling, and then performs an additive skip connection operation with the visual features of the encoder at the same level. Subsequently, the visual features and the text features of the dual-granularity semantic cues are processed by the text-guided spatial attention TGSA module to generate spatial attention weights and are weighted in a residual manner. After passing through five levels of decoding blocks, the final output is a segmentation mask with the same resolution as the original image. The TGSA module generates spatial attention weights through the following steps: Step a): The fused visual features and text features are respectively mapped to the shared latent space through a multilayer perceptron; Step b): Calculate the spatial similarity between visual features and text features through dot product operation, and obtain the attention weights by Softmax normalization; Step c): Weight the attention weights to the fused features using a residual approach; S3: Input the preprocessed infrared image into the target detection network constructed in step S2 for network training to obtain a trained network; wherein, the network is jointly trained in an end-to-end manner, the text encoder parameters are frozen during the training process, the image-text inversion network does not share gradient information with other networks, and is optimized separately using contrast loss, wherein the contrast loss is the contrast loss between the result of the image-text inversion network and the visual features. S4: After preprocessing the single infrared image to be detected according to step S1, input it into the trained target detection network, and postprocess the output segmentation mask. The resulting mask is the final target detection result.

[0007] Specifically, the image encoder described in step S2 adopts a five-level UNet structure, with each level followed by a channel attention layer.

[0008] Specifically, the fixed template text mentioned in step S2 is "a photo of a s1* in an infrared image, with s2* targets in the sky, gound or ocean.", and the text encoder is a CLIP-RN50 with frozen weights, where * represents a placeholder.

[0009] Specifically, the image-text inversion network described in step S2 consists of cross-attention and a multilayer perceptron, and its output dimension is consistent with that of the text encoder.

[0010] Specifically, the post-processing of the output segmentation mask in step S4 includes ensuring it maintains the same resolution as the input image, with pixel values ​​ranging from [0,1], and performing binarization with a threshold of 0.5. Further, the following joint loss function is used during network training in step S3: L = λ1L BCE + λ2L SoftIoU + λ3L contra (1) Where L represents the total loss, L BCE For binary cross-entropy loss, L SoftIoU For Soft-IoU loss, L contra λ1, λ2, and λ3 represent the visual-text contrast loss, and λ1, λ2, and λ3 are the corresponding balance coefficients.

[0011] The beneficial effects of this invention are as follows: By adopting a semantic prompting strategy of "coarse-grained fixed template + fine-grained dynamic text inversion," this invention can provide scene-level and instance-level dual semantic priors for infrared images without relying on any manual text annotation, significantly reducing deployment costs and effectively alleviating the false detection and missed detection problems of traditional methods under complex backgrounds and low signal-to-noise ratio conditions. Specifically, the coarse-grained prompts provide the network with stable scene context priors, while the fine-grained prompts capture the unique semantic features of the current image in real time through the text inversion network. The two complement each other, enabling the network to maintain high discriminative power even in extreme scenarios such as missing target textures, blurred shapes, and strong background clutter. The text-guided spatial attention (TGSA) module directly maps high-level semantics to the spatial attention map, achieving precise focusing on potential target regions and effective suppression of background noise, significantly improving detection confidence and reducing false alarm rates. The end-to-end training and inference process requires no additional text input or post-processing parameter adjustment, and can be directly embedded into airborne, missile-borne, spaceborne, and edge computing devices, achieving "plug-and-play" deployment. Attached Figure Description

[0012] Figure 1 This is a flowchart of an infrared weak target detection method with dual-granularity semantic prompts according to the present invention; Figure 2 This is a schematic diagram of the target detection network structure based on dual-granularity semantic prompting of the present invention; Figure 3 This is a structural block diagram of the image encoders of each layer in the embodiments of the present invention; Figure 4 This is a structural block diagram of each layer of the image decoder in the embodiments of the present invention; Figure 5 This is a structural block diagram of the spatial attention module in an embodiment of the present invention. Detailed Implementation

[0013] The present invention will be further described below with reference to the accompanying drawings and embodiments. The present invention includes, but is not limited to, the following embodiments.

[0014] The implementation environment was a server with four NVIDIA RTX 4090D graphics cards (24 GB), running Ubuntu 20.04 and Python version 3.8. The experiments used three public datasets: NUAA-SIRST, NUDT-SIRST, and IRSTD-1K, all with a uniform resolution of 256×256, and a training / test ratio of 5:5 or 8:2. This invention provides a dual-granularity semantic cueing-based infrared weak target detection method, such as... Figure 1 As shown, the specific implementation process is as follows: S1: Preprocess the input single infrared image to obtain a standardized infrared image.

[0015] S2: Construct an object detection network based on dual-granularity semantic prompts, such as Figure 2 As shown, it mainly includes an image encoder, an image-text inversion network, and a text-guided image decoder.

[0016] The image encoder employs a five-level UNet structure with channel attention, with each level followed by a channel attention layer to suppress background noise and enhance the features of small targets. For example... Figure 3 As shown, each layer of the image encoder first performs downsampling on the image features using a max-pooling layer; then, two consecutive convolutional modules extract the features, each containing a convolutional layer, a batch normalization layer, and a ReLU activation layer; the extracted features are then fed into a channel attention layer for enhancement. The image encoder performs multi-level feature extraction on a standardized infrared image to obtain multi-scale visual features. The multi-scale visual features output by the encoder are denoted as f. i (i=1,…,5), where high-level visual features f 5 It will be fed into an image-text inversion network to adaptively and dynamically generate fine-grained semantic information corresponding to the image.

[0017] The image-text inversion network consists of cross-attention and a multilayer perceptron, outputting a semantic vector with the same dimension as the text encoder to ensure cross-modal alignment. This vector is used to replace placeholder semantic blocks in the coarse-grained semantic information. The image-text inversion network and the text encoder constitute a dual-granularity semantic prompting module, generating dual-granularity semantic prompts corresponding to the input image. The text encoder uses a CLIP-RN50 model with frozen weights, which encodes a fixed coarse-grained text template "a photo of a s1* in an infrared image, with s2* targets in the sky, ground, or ocean.", outputting a 77×512 semantic vector. After integrating the fine-grained semantic information learned by the inversion network into this vector, the dual-granularity semantic prompts are obtained.

[0018] Text-guided image decoding is performed as a five-level cascaded structure. Before this cascaded structure, a cross-attention mechanism is first used to process the high-level visual features f. 5 It integrates with dual-granularity semantic cues to achieve interaction between global visual features and semantic features. Figure 4The block diagram of each layer of the image decoder is given. At each level of the decoder, bilinear interpolation is first used for upsampling, followed by a skip connection operation that adds the visual features of the same level encoder. Subsequently, the visual features and the text features of the dual-granularity semantic cues are processed by the Text-Guided Spatial Attention (TGSA) module to generate spatial attention weights, which are then weighted in a residual manner. After passing through five consecutive decoding blocks, the final output is a segmentation mask with the same resolution as the original image.

[0019] The structural block diagram of the TGSA module is as follows: Figure 5 As shown, it generates spatial attention weights through the following steps: Step a): The fused visual features and text features are respectively mapped to the shared latent space through a multilayer perceptron; Step b): Calculate the spatial similarity between visual features and text features through dot product operation, and obtain the attention weights by Softmax normalization; Step c): The attention weights are weighted to the fusion features in a residual manner to achieve target region enhancement and background suppression.

[0020] S3: Input the preprocessed infrared image into the target detection network constructed in step S2, and train the network in an end-to-end manner to obtain the trained network. During training, the text encoder parameters are frozen, and the image-to-text inversion network does not share gradient information with other networks, but is optimized independently using a contrastive loss method. The loss function uses a joint loss: L = λ1L BCE + λ2L SoftIoU + λ3L contra (2) Where L represents the total loss, L BCE For binary cross-entropy loss, L SoftIoU For Soft-IoU loss, L contra The visual-text contrastive loss is calculated by comparing the results of the image-text inversion network (i.e., the fused dual-granularity text semantic cues) with visual features to constrain the learning of the inversion network; λ1, λ2, and λ3 are the corresponding balance coefficients. The AdamW optimizer is used during training, with an initial learning rate of 1e-4, a batch size of 4, and 800 training epochs.

[0021] S4: After preprocessing the single infrared image to be detected as in step S1, it is input into the trained target detection network (i.e., the inference stage). First, the image encoder and the image-text inversion network jointly generate semantic cues. Then, the text-guided image decoder uses image features at each level and semantic cues to output a segmentation mask. The mask is binarized with a threshold of 0.5 to obtain the final output. The resulting binary mask is the final target detection result. The entire inference process requires no additional text input or manual post-processing.

[0022] The infrared weak target detection system with dual-granularity semantic prompts provided by this invention embeds both coarse-grained scene priors and fine-grained instance semantics into the network simultaneously. During the decoding process, high-level semantics are explicitly mapped to spatial attention weights through text-guided spatial attention. As a result, the system can maintain high discriminative power even under conditions of missing target texture, extremely low signal-to-noise ratio, and severe background clutter, significantly reducing false alarms and missed detections. At the same time, the text inversion network automatically learns the visual-language mapping during the training phase, and no manual text or external prompts are required during the inference phase, enabling "plug-and-play" deployment and greatly reducing the cost of engineering implementation.

[0023] Experimental results show that the system achieves leading IoU and detection probability on three public datasets—NUAA-SIRST, NUDT-SIRST, and IRSTD-1K—with a low input resolution of 256×256. Furthermore, the system has a small overall parameter count, allowing it to run directly on airborne, missile-borne, spaceborne, and edge computing devices. It boasts advantages such as high accuracy, robustness, and good portability, making it highly suitable for infrared target detection tasks with stringent requirements for size, power consumption, and real-time performance, such as early warning detection, precision guidance, and disaster search and rescue.

Claims

1. A method for detecting weak infrared targets with dual-granularity semantic cues, characterized in that... The steps are as follows: S1: Preprocess the input single infrared image to obtain a standardized infrared image; S2: Construct a target detection network based on dual-granularity semantic prompts, mainly including an image encoder, an image-text inversion network, and a text-guided image decoder; The image encoder performs multi-level feature extraction on the input image to obtain multi-scale visual features; The image-text inversion network and text encoder constitute a dual-granularity semantic prompt module, generating dual-granularity semantic prompts corresponding to the input image. Specifically, a text encoder with fixed template text and frozen parameters is used to obtain coarse-grained semantic features, and an image-text inversion network is used to map visual features to dynamically generate fine-grained semantic features. The coarse-grained semantic features and fine-grained semantic features constitute dual-granularity semantic prompts. The text-guided image decoder progressively fuses multi-scale visual features with dual-granularity semantic cues and outputs a segmentation mask for small infrared targets. The text-guided image decoder is a five-level cascaded structure. Before the first level, a cross-attention module fuses high-level visual features and dual-granularity semantic cues. Each of the remaining levels includes upsampling and a text-guided spatial attention TGSA module. The specific processing is as follows: each level of the decoder first performs bilinear interpolation upsampling, and then performs an additive skip connection operation with the visual features of the encoder at the same level. Subsequently, the visual features and the text features of the dual-granularity semantic cues are processed by the text-guided spatial attention TGSA module to generate spatial attention weights and are weighted in a residual manner. After passing through five levels of decoding blocks, the final output is a segmentation mask with the same resolution as the original image. The TGSA module generates spatial attention weights through the following steps: Step a): The fused visual features and text features are respectively mapped to the shared latent space through a multilayer perceptron; Step b): Calculate the spatial similarity between visual features and text features through dot product operation, and obtain the attention weights by Softmax normalization; Step c): Weight the attention weights to the fused features using a residual approach; S3: Input the preprocessed infrared image into the target detection network constructed in step S2 for network training to obtain a trained network; wherein, the network is jointly trained in an end-to-end manner, the text encoder parameters are frozen during the training process, the image-text inversion network does not share gradient information with other networks, and is optimized separately using contrast loss, wherein the contrast loss is the contrast loss between the result of the image-text inversion network and the visual features. S4: After preprocessing the single infrared image to be detected according to step S1, input it into the trained target detection network, and postprocess the output segmentation mask. The resulting mask is the final target detection result.

2. The infrared weak target detection method with dual-granularity semantic cues as described in claim 1, characterized in that: The image encoder described in step S2 adopts a five-level UNet structure, with each level followed by a channel attention layer.

3. The infrared weak target detection method with dual-granularity semantic cues as described in claim 1, characterized in that: The fixed template text mentioned in step S2 is "a photo of a s1* in an infrared image, with s2* targets in the sky, gound or ocean." The text encoder is a CLIP-RN50 with frozen weights, where * represents a placeholder.

4. The infrared weak target detection method with dual-granularity semantic cues as described in claim 1, characterized in that: The image-text inversion network described in step S2 consists of cross-attention and a multilayer perceptron, and its output dimension is consistent with that of the text encoder.

5. The infrared weak target detection method with dual-granularity semantic cues as described in claim 1, characterized in that: The post-processing of the output segmentation mask described in step S4 includes making it consistent with the resolution of the input image, with pixel values ​​ranging from [0,1], and performing binarization with a threshold of 0.

5.

6. The infrared weak target detection method with dual-granularity semantic cues as described in claim 1, characterized in that: The following joint loss function is used during network training in step S3: L = λ1 L BCE + λ2 L SoftIoU + λ3 L contra Where L represents the total loss, L BCE For binary cross-entropy loss, L SoftIoU For Soft-IoU loss, L contra λ1, λ2, and λ3 represent the visual-text contrast loss, and λ1, λ2, and λ3 are the corresponding balance coefficients.

Images