An infrared image target detection method based on image text mask joint learning

Abstract

This invention relates to the technical field of target detection, and discloses an infrared image target detection method based on joint learning of image, text, and mask. The method includes constructing a recognition network model, which comprises parallel image, text, and mask branches. The image branch includes an image backbone network, a cross-modal fusion neck network, and a detection head. During training, text embedding vectors and mask embedding vectors are first constructed from the text and mask branches, respectively. Simultaneously, multi-scale image features are extracted from the image backbone network. Then, combined with the total loss function, the cross-modal fusion neck network is used to fuse the multi-scale image features, text embedding vectors, and mask embedding vectors to obtain fused features. These fused features, text embedding vectors, and mask embedding vectors are then input into the detection head to achieve target detection, completing model training. During inference, only the image branch of the trained recognition network model is used for inference detection.

Description

Technical Field

[0001] This invention belongs to the technical field of target detection, specifically relating to an infrared image target detection method based on joint learning of image and text masks. Background Technology

[0002] Infrared imaging technology has the advantages of clear imaging under no lighting conditions and penetrating atmospheric interference such as fog and smoke. It plays an irreplaceable role in target detection in day and night and in adverse weather conditions, and has become an important research direction in the field of computer vision.

[0003] Existing infrared target detection methods are mainly divided into traditional model-driven methods and deep learning data-driven methods. Traditional methods include those based on filters, the human visual system (HVS), and low-rank representations. These methods rely on manually designed features and hyperparameters, exhibit poor robustness in complex backgrounds and low signal-to-noise ratio (SNR) scenes, and require specialized domain knowledge. While deep learning methods have made significant progress and can be categorized into general target detection frameworks, segmentation techniques, and generative adversarial networks (GANs), they typically rely on a single type of image data, resulting in limited generalization ability of the models in complex scenes. With the development of vision-language models, many research methods have incorporated linguistic information into the field of target detection. However, infrared targets, due to their different imaging mechanisms compared to visible light, generally suffer from low brightness and a lack of texture and shape features, making it difficult to accurately describe the target's own characteristics using language.

[0004] Therefore, existing infrared target detection methods still need improvement in terms of adaptability to complex infrared scenes, detection accuracy, and generalization ability. There is an urgent need for an infrared target detection method that is more applicable and more accurate. Summary of the Invention

[0005] This invention provides an infrared image target detection method based on joint learning of image and text masks, which solves the problems of insufficient feature fusion and low target detection accuracy in existing infrared target detection methods, and improves the accuracy and robustness of infrared target detection.

[0006] To achieve the above objectives, the present invention provides the following technical solution: An infrared image target detection method based on joint learning of image, text, and mask is proposed. The method constructs a recognition network model, which includes parallel image, text, and mask branches. The image branch comprises an image backbone network, a cross-modal fusion neck network, and a detection head. During training, text embedding vectors and mask embedding vectors are first constructed by the text branch and mask branch, respectively. At the same time, multi-scale image features are extracted by the image backbone network. Then, combined with the total loss function, the cross-modal fusion neck network is used to fuse the multi-scale image features, text embedding vectors, and mask embedding vectors to obtain fused features. Finally, the fused features, text embedding vectors, and mask embedding vectors are input into the detection head to achieve target detection and complete model training. During inference, only the image branches in the pre-trained recognition network model are used for inference detection.

[0007] Furthermore, the cross-modal fusion neck network adopts a top-down and bottom-up bidirectional fusion network structure, including two text-guided C3k2 modules (TGC3k2 group), a text-enhanced attention module, and a hierarchical attention fusion module (HAFB group). The first TGC3k2 group takes multi-scale image features and text embedding vectors as input, and passes high-level semantic information to low-level information through a top-down path. Its output is horizontally connected to the second TGC3k2 group and also connected to the text enhancement attention module. The text enhancement attention model then updates the text embedding vectors by combining the mask embedding vectors. The second TGC3k2 group takes the output of the first TGC3k2 group and the updated text embedding vector as input, and passes the low-level detailed information to the high-level through a bottom-up path. Its output is horizontally connected to the HAFB group, which then combines the mask embedding vector and multi-scale image features to perform feature fusion and obtain fused features.

[0008] Furthermore, for each text-guided C3k2 module in the first TGC3k2 group, the high-level features are first downsampled to make their resolution the same, and then the two are concatenated. The concatenated image features After CBS processing, the image features are segmented into two parts according to the channel dimension. , Then image features The image features are processed by a series-connected C3k module. Then, the image features Together with the text embedding vector, the image is processed by the TG attention module to obtain image features. Finally, the image features and image features After performing a splicing operation and then processing it through CBS, the output features are obtained. ; For each text-guided C3k2 module in the second TGC3k2 group, the low-level features need to be upsampled to make the resolution of the current layer features the same, then concatenated, and the updated text embedding vector is sent to the TG attention module for processing. The remaining operations are the same as those in the text-guided C3k2 module in the first TGC3k2 group.

[0009] Furthermore, the text enhancement attention module first guides the output features of the C3k2 module to each text in the first TGC3k2 group. The text embedding vector is pooled separately with the mask embedding vector, and then the pooled features are concatenated. Finally, the text embedding vector is used as the query vector, and the concatenated features are used as the key vector and value vector. The text embedding vector is updated using a cross-attention mechanism.

[0010] Furthermore, the hierarchical attention fusion module adopts a symmetrical parallel structure design, including two sets of local branches and global branches, which are symmetrically distributed relative to the convolution branches. The first set of local and global branches takes the current layer features from the multi-scale image features as input, and processes them through LocalGlobalAttention attention modules with different patch parameters to obtain local features. and global features Then, the two are concatenated and output. The second set of local and global branches uses the output features of the current layer of the second set of TGC3k2 as input, and processes them through LocalGlobalAttention modules with different patch parameters to obtain local features. and global features Then, the two are concatenated and output. The convolutional branch takes the mask embedding vector as input, first adds it to the current layer features in the multi-scale image features and the output features of the current layer in the second TGC3k2 group, then concatenates it with the concatenated outputs of the two local branches and the global branches, and finally outputs the fused features through convolution.

[0011] Furthermore, the multi-scale image features are set as image features P2-P5. Both the first and second TGC3k2 groups have three text-guided C3k2 modules: a top-level module, a middle-level module, and a bottom-level module. The HAFB group has three hierarchical attention fusion modules. The image features P2-P4 are respectively connected to the input terminals of the three text-guided C3k2 modules of the first TGC3k2 group, the output terminals of the three text-guided C3k2 modules of the second TGC3k2 group are respectively connected to the input terminals of the three hierarchical attention fusion modules, the image features P3-P5 are also respectively connected to the input terminals of the three hierarchical attention fusion modules, and the output terminals of the three hierarchical attention fusion modules are all connected to the detection head; The image feature P5 is simultaneously connected to the top-level modules of the first TGC3k2 group and the second TGC3k2 group. The middle-level and bottom-level modules of the second TGC3k2 group are respectively connected to the top-level and middle-level modules of the first TGC3k2 group. The bottom-level module of the first TGC3k2 group is simultaneously connected to the bottom-level module and the detection head of the second TGC3k2 group.

[0012] Furthermore, the masking branch first processes the infrared image using the Segment Anything Model 3 (SAM3) to generate a target mask image, and then uses a mask encoder to process the target mask image and multi-scale image features to obtain a mask embedding vector. The text branch uses the Generative Pretraining Transformer (GPT) natural language model to process the infrared image, generating a text description containing target location, quantity, and background information, and then uses the CLIP text encoder to construct a text embedding vector.

[0013] Furthermore, the total loss function is calculated using the following formula. , in, This represents the predicted bounding box brightness value. This represents the brightness value of the actual bounding box. Let B represent the pixel-level difference, B represent the set of adjusted bounding boxes, and N represent the number of pixels in each bounding box. Indicates the distribution focus loss. and These represent IoU loss respectively. and brightness difference loss The weighting coefficients, and the sum of the two is 1. This represents the target-text contrast loss function. This represents the target-mask contrast loss function.

[0014] A computing storage medium for storing a computer program, which, when executed by a processor, causes an electronic device to perform the infrared image target detection method based on image-text mask joint learning as described above.

[0015] An electronic device includes a memory, a processor, and a computer program stored within the memory. When the processor executes the computer program, it causes the electronic device to perform the infrared image target detection method based on image-text mask joint learning as described above.

[0016] Compared with the prior art, the beneficial effects of the present invention are: (1) The present invention designs an infrared image target detection method based on joint learning of image, text and mask. On the basis of conventional image branch, a text branch and a mask branch are added. The text modal branch assists the visual network in identifying the target region by constructing semantic context information. At the same time, the mask modal branch uses a mask encoder to generate target mask prompt information, providing the target's own features to guide the network to accurately locate the target, making the detection process more targeted and effective.

[0017] (2) In order to fully enhance the interaction between image, text and mask features, the present invention designs a text-guided C3K2 module to integrate text information into multi-scale image features, and a text enhancement attention module uses image and mask features to update text embedding vectors to fuse multimodal information, thereby improving the visual semantic representation capability.

[0018] (3) The present invention designs a hierarchical attention fusion module, which uses a hierarchical attention mechanism to process local and global features to achieve adaptive feature fusion, while enhancing cross-channel information transmission and improving the model's ability to perceive complex scenes. In addition, in the target detection head, the present invention designs a brightness difference loss function by exploring the brightness distribution difference law of the infrared small target anchor frame of the model, thereby improving the positioning accuracy of infrared small targets. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall structure of the recognition network model of the present invention; Figure 2 This is a schematic diagram of the mask encoder of the present invention; Figure 3 This is a schematic diagram of the text guidance C3k2 module of the present invention; Figure 4 This is a schematic diagram of the text feature enhancement attention module of the present invention; Figure 5 This is a schematic diagram of the hierarchical attention fusion module of the present invention; Figure 6The image shows a comparison of the detection results of the target detection method of the present invention with those of other methods, where blue circles indicate missed detections and yellow circles indicate false detections. Detailed Implementation

[0020] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the infrared image target detection method based on image-text mask joint learning of this invention. It should be noted that the description of these embodiments is for the purpose of helping to understand this invention, but does not constitute a limitation of this invention.

[0021] This invention provides an infrared target detection method based on image-text-mask joint learning, such as... Figure 1 As shown, the recognition network model used consists of three parallel branches: an image branch, a text branch, and a mask branch. The image branch includes a cascaded image backbone network, a cross-modal fusion neck network, and a detection head.

[0022] During training, the training set is input into the recognition network model. First, the image backbone network in the image branch uses the Darknet network of the YOLO model to extract multi-scale image features {P2, P3, P4, P5}. The text branch uses a pre-trained CLIP text encoder to represent and encode the text description, constructing semantic text embedding vectors to supplement the image detection network with rich contextual semantic information. The mask branch extracts features from the visual mask through a mask encoder, generating target perception attention weights and semantic features to construct mask embedding vectors, thereby guiding the model to accurately locate and focus on the target region. Then, combined with a customized total loss function, the image branch's cross-modal fusion neck network (CMFNeck) achieves cross-modal feature fusion of multi-scale image features, text embedding vectors, and mask embedding vectors. Finally, the image branch's detection head achieves target detection, completing the training of the recognition network model. During inference, the text branch and mask branch of the trained recognition network model no longer participate in the calculation; only the image branch is used to perform forward inference to achieve object detection.

[0023] Specifically as follows: Step 1. Acquire multi-source data, including infrared images, text embedding vectors, and mask embedding vectors.

[0024] 1. Acquire infrared images 2. Generate text embedding vectors for targets in infrared images to achieve text descriptions. Based on the infrared images, we use Generative Pretraining Transformer (GPT) to generate text descriptions that include target location, quantity, and background information.

[0025] Specifically, GPT is required to concisely describe the main elements of an image, focusing on the spatial location and overall quantity of targets, as well as the surrounding background. We first assign GPT the input instruction by assigning it the role of an expert in identifying and locating small targets in infrared images, and the output is a concise text prompt description. Then, a pre-trained CLIP text encoder is used to encode the text prompt description to obtain a text embedding vector that accurately summarizes the key information conveyed by the image.

[0026] 3. Generate the target mask image from the infrared image, and then generate the mask embedding vector. First, based on the infrared image, we use Segment Anything Model 3 (SAM3) to generate the target mask image. Specifically, we combine the input image with the target detection bounding box coordinate information (x is the x-coordinate of the top-left corner of the bounding box, y is the y-coordinate of the top-left corner of the bounding box, w is the width of the bounding box, and h is the height of the bounding box) to obtain the ground truth image. Then, we feed the target anchor boxes from the ground truth image into SAM3 to obtain the target mask image, i.e., the mask hint.

[0027] Then, the target mask image is feature extracted by the mask encoder to generate target perception attention weights and semantic features to construct the mask embedding vector, thereby guiding the model to achieve accurate localization and focusing of the target region.

[0028] Specifically, such as Figure 2 As shown, the mask hint is first processed by the CBS module using a mask encoder to obtain the mask hint feature. Meanwhile, we processed the multi-scale image features {P2, P3, P4, P5} generated from the feature pyramid PAN through the first group of CBS modules and upsampling operations to obtain image features. Then, the mask hints at the features. Image features The data is then stitched together and processed by the CBS module to generate target perception weights. Simultaneously, the multi-scale image features {P2, P3, P4, P5} are processed through the second set of CBS modules to obtain the corresponding semantic features {S2, S3, S4, S5}. Finally, we apply the target perception weights... The mask embedding vector is obtained by aggregating the semantic features {S2, S3, S4, S5}. : in, is the output mask embedding vector, softmax is the normalization operation, and U is the upsampling operation.

[0029] Step 2. Use a cross-modal fusion neck network to perform feature fusion, obtain fused features, and then combine the fused features with text embedding vectors and mask embedding vectors to input the detection head to achieve target detection.

[0030] 1. This cross-modal fusion neck network (CMFNeck) follows a bidirectional path of top-down and bottom-up convergence networks. It is used to cross-modal fuse multi-scale image features, text embedding vectors, and mask embedding vectors to simultaneously enhance the quality of feature extraction. It includes two text-guided C3k2 modules (TGC3k2 group), a text-enhancing attention module (TEAttention) and a hierarchical attention fusion module (HAFB group). The three core modules are: the text-guided C3k2 module (TGC3k2), the text-enhancing attention module (TEAttention), and the hierarchical attention fusion module (HAFB), as shown below. Figure 3 , 4 As shown in Figure 5.

[0031] The first TGC3k2 group takes multi-scale image features and text embedding vectors as input, and passes high-level semantic information to low-level layers through a top-down path. Its output is horizontally connected to the second TGC3k2 group and also connected to the text enhancement attention module TEAttention. The TEAttention model then updates the text embedding vectors by combining the mask embedding vectors. The second TGC3k2 group takes the output of the first TGC3k2 group and the updated text embedding vector as input. It passes the low-level detailed information to the high-level through a bottom-up path. Its output is horizontally connected to the HAFB group. The HAFB group then combines the mask embedding vector and multi-scale image features to perform feature fusion and obtain fused features.

[0032] Specifically, both the first and second TGC3k2 groups have three text-guided C3k2 modules: a top-level module, a middle-level module, and a bottom-level module. The HAFB group has three levels of attention fusion modules. The image features P2-P4 are respectively connected to the input terminals of the three text-guided C3k2 modules of the first TGC3k2 group, the output terminals of the three text-guided C3k2 modules of the second TGC3k2 group are respectively connected to the input terminals of the three hierarchical attention fusion modules, the image features P3-P5 are also respectively connected to the input terminals of the three hierarchical attention fusion modules, and the output terminals of the three hierarchical attention fusion modules are all connected to the detection head; The image feature P5 is simultaneously connected to the top-level modules of the first TGC3k2 group and the second TGC3k2 group. The middle-level and bottom-level modules of the second TGC3k2 group are respectively connected to the top-level and middle-level modules of the first TGC3k2 group. The bottom-level module of the first TGC3k2 group is simultaneously connected to the bottom-level module and the detection head of the second TGC3k2 group.

[0033] I. Text-guided C3K2 module TGC3k2 For each text-guided C3k2 module in the first TGC3k2 group, the high-level features are first downsampled to make their resolution the same, and then the two are concatenated. The concatenated image features After CBS processing, the image features are segmented into two parts according to the channel dimension. , Then image features The image features are processed by a series-connected C3k module. Then, the image features Together with the text embedding vector, the image is processed by the TG attention module to obtain image features. Finally, the image features and image features After performing a splicing operation and then processing it through CBS, the output features are obtained. ; For each text-guided C3k2 module in the second TGC3k2 group, the low-level features need to be upsampled to make the resolution of the current layer features the same, then concatenated, and the updated text embedding vector is sent to the TG attention module for processing. The remaining operations are the same as those in the text-guided C3k2 module in the first TGC3k2 group.

[0034] Specifically, such as Figure 3 As shown, given a text embedding vector and image features The text-guided C3K2 module (TGC3K2) first extracts image features. After CBS processing, the image features are segmented according to the channel dimension. , and image features Image features are obtained by feeding them into n cascaded C3K modules. Finally, the TG attention module (TGAttnBlock) is used to calculate the embedding vector with the text. Similarity, weighted fusion of text and image features: in, This is the output of the TGC3K2 module. This is the output of the TG attention module. This represents the sigmoid function. This is the output of the CBS module.

[0035] II. Text Feature Enhancement Attention Module TEAttention After fusing image information with text embedding vectors, in order to enhance text embedding with image information and mask embedding, we aggregate image and mask features and update text embedding through the text feature enhancement attention module TEAttention.

[0036] Specifically, such as Figure 4 As shown, firstly, the texts in the first TGC3k2 group are used to guide the output features of the C3k2 module. That is, the multi-scale features (L2, L3, L4, L5) are respectively coupled with the mask embedding vector. Perform feature pooling, and then concatenate the pooled features: in, The features of the stitched image, The pooled image features are the i-th layer features, and Pooling is the max pooling operation.

[0037] Finally, we embed the text vector as query vector Features of the stitched image as a key vector Sum value vector The text embedding vector is updated using a cross-attention mechanism: in, The updated text embedding vector is obtained by adding the enhanced text embedding vector to the original text embedding vector. for Dimensions.

[0038] III. Hierarchical Attention Fusion Module (HAFB) This layer of attention fusion module adopts a symmetrical parallel structure design, including two sets of local branches and global branches, which are symmetrically distributed relative to the convolutional branches. It divides feature patches through the LocalGlobalAttention attention mechanism and combines cosine similarity to filter relevant features, thereby achieving adaptive fusion of local and global features.

[0039] The first set of local and global branches takes the current layer features from the multi-scale image features as input, and processes them through LocalGlobalAttention attention modules with different patch parameters to obtain local features. and global features Then, the two are concatenated and output. The second set of local and global branches uses the output features of the current layer of the second set of TGC3k2 as input, and processes them through LocalGlobalAttention modules with different patch parameters to obtain local features. and global features Then, the two are concatenated and output. The convolutional branch takes the mask embedding vector as input, first adds it to the current layer features in the multi-scale image features and the output features of the current layer in the second TGC3k2 group, then concatenates it with the concatenated outputs of the two local branches and the global branch, and finally outputs the fused features through convolution.

[0040] Specifically, such as Figure 5 As shown, given the current layer image features output by the second TGC3k2 group... Image features of the current layer output by the Image Pyramid PAN First, image features After undergoing dimensionality reduction through 1×1 convolutions, the data is fed into the LocalGlobalAttention module for further processing to obtain local features. and global features At the same time, image features After undergoing dimensionality reduction through 1×1 convolutions, the data is fed into the LocalGlobalAttention module for further processing to obtain local features. and global features Furthermore, the features after 1×1 convolution dimensionality reduction are embedded into the mask embedding vector. The features are then obtained by performing addition and fusion processing, followed by a 3×3 convolution operation. Output; Then, we construct local and global branches by controlling the patch size parameter p of the LocalGlobalAttention module. This parameter divides the feature map into non-overlapping patches in the spatial dimension. When p=2, it means that the module extracts local features, and when p=4, the module extracts global features. Finally, we will extract two local features using the LocalGlobalAttention module. ( The two global features are concatenated. ( The concatenation is then combined with the convolutional features. Fine-grained fusion features are obtained by performing splicing and convolution operations. .

[0041] in, ( ), ( These represent local features and global features, respectively. It is the LocalGlobalAttention module. and These are ordinary convolution and reparameterized convolution operations, respectively.

[0042] 2. Detection head The detection head adopts a vision-text collaborative detection head structure, which consists of two parts: a target comparison detection branch and a regression branch.

[0043] Specifically, the target contrast detection branch replaces the traditional classification branch. For text and mask embedding vectors, we refer to the YOLO-World[1] by Cheng et al. and the YOLOE[2] by Wang et al. to construct target-text contrast loss functions in the target contrast head respectively. Comparison of target-mask loss function To optimize model training.

[0044] In the regression branch, the method of this invention designs a brightness difference loss function. (BDLoss) improves localization accuracy by extracting the image content of the predicted bounding box and the ground truth label bounding box, normalizing the pixel intensity, and then calculating the brightness difference pixel by pixel. The calculation formula is as follows: in, This represents the predicted bounding box brightness value. This represents the brightness value of the actual bounding box. This represents pixel-level differences, where B represents the set of adjusted bounding boxes, and N represents the number of pixels in each bounding box.

[0045] The total loss function is composed of regression loss Loss composition compared to target Together they constitute: in, Indicates the distribution focus loss. and These represent IoU loss respectively. and brightness difference loss The weighting coefficients, and the sum of the two is 1. This represents the target-text contrast loss function. This represents the target-mask contrast loss function.

[0046] Step 3. Identify the training and inference of the network model. Model training The aforementioned recognition network model was implemented using PyTorch and executed on an NVIDIA GeForce RTX 4090. The initial learning rate was 0.001, and the selected optimizer was SGD, optimized with a momentum of 0.937 to ensure optimal performance. The dataset was divided into training and test sets in a 7:3 ratio, and all images were uniformly resized to 640×640 pixels. After the recognition network model training was complete, the program automatically closed, generating infrared image detection results and saving them locally.

[0047] Object detection inference During the inference phase, the trained network only uses the image branch to perform forward inference; the text and mask branches are no longer involved in the calculation. The preprocessed image to be detected is input into the trained recognition network model, and processed using the image branch. After feature extraction, multimodal fusion, and detection head calculation, the final output is the detection result of the infrared target, including the target bounding box coordinates and detection confidence.

[0048] The present invention also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, causes an electronic device to perform the infrared image target detection method based on image-text mask joint learning as described above.

[0049] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein when the processor executes the computer program, the electronic device performs the infrared image target detection method based on image-text mask joint learning as described above.

[0050] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0051] The method steps in the embodiments of this application can be implemented by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can be located in an ASIC. Additionally, the ASIC can be located in a terminal device. Alternatively, the processor and storage medium can exist as discrete components in the terminal device.

[0052] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the above embodiments are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0053] To verify the feasibility of the target detection method of the present invention, we conducted the following experiments: This invention uses precision (P), recall (R), F1 score, and COCO precision (AP50, AP50:95) as core evaluation indicators, where the IoU threshold for P and R is 0.5.

[0054] The effectiveness of this invention is demonstrated by comparisons with several excellent object detection algorithms on a test set of the constructed dataset. Specifically, we compare our proposed method with seven CNN detection methods and five Transformer detection methods. Our proposed method outperforms the second-place method by 2.9, 1.2, 2.2, 1.7, and 18.3 in precision (P), recall (R), F1 score, AP50, and AP50:95, respectively.

[0055] Table 1 Comparison of quantitative results from different methods A comparison of the qualitative results of the method of this invention with those of other methods is as follows: Figure 6As shown, this dataset displays five images from different scenes, each labeled with a corresponding real-world ship tag. These images include dimly lit, near-shore, and moored infrared targets of small ships, as well as numerous interfering elements such as clouds, islands, and buildings. Furthermore, they cover different time periods, including day and night. Each row shows the results of different methods, with yellow circles indicating false positives and blue circles indicating false negatives.

[0056] Traditional methods often misidentify shoreline structures and islands as ships, and are also affected by cloud cover and sea clutter, with many methods only identifying a portion of moored vessels. In contrast, the method of this invention, through multimodal joint learning, can accurately detect these small infrared targets in different scenarios and time periods, demonstrating significant robustness in challenging environments.

[0057] The quantitative and qualitative experimental results above demonstrate that our proposed image-text-mask joint learning infrared target detection network model, i.e., the recognition network model, successfully solves the problem of low feature fusion efficiency in existing infrared target detection methods through text and mask learning and cross-modal fusion, thereby improving the accuracy of infrared target detection.

[0058] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples. Various changes or modifications can be made to these embodiments without departing from the principles and essence of the present invention. Therefore, the scope of protection of the present invention is defined by the appended claims.

[0059] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Various modifications or variations that can be made by those skilled in the art without creative effort within the scope of the appended claims are still within the scope of protection of the present invention.

Claims

1. An infrared image target detection method based on joint learning of image and text masks, characterized in that: A recognition network model is constructed, comprising parallel image branches, text branches, and mask branches. The image branch includes an image backbone network, a cross-modal fusion neck network, and a detection head. During training, text embedding vectors and mask embedding vectors are first constructed by the text branch and mask branch, respectively. At the same time, multi-scale image features are extracted by the image backbone network. Then, combined with the total loss function, the cross-modal fusion neck network is used to fuse the multi-scale image features, text embedding vectors, and mask embedding vectors to obtain fused features. Finally, the fused features, text embedding vectors, and mask embedding vectors are input into the detection head to achieve target detection and complete model training. During inference, only the image branches in the pre-trained recognition network model are used for inference detection.

2. The infrared image target detection method based on joint learning of image and text masks according to claim 1, characterized in that: The cross-modal fusion neck network adopts a top-down and bottom-up bidirectional fusion network structure, including two text-guided C3k2 modules (TGC3k2 group), a text-enhanced attention module, and a hierarchical attention fusion module (HAFB group). The first TGC3k2 group takes multi-scale image features and text embedding vectors as input, and passes high-level semantic information to low-level information through a top-down path. Its output is horizontally connected to the second TGC3k2 group and also connected to the text enhancement attention module. The text enhancement attention model then updates the text embedding vectors by combining the mask embedding vectors. The second TGC3k2 group takes the output of the first TGC3k2 group and the updated text embedding vector as input, and passes the low-level detailed information to the high-level through a bottom-up path. Its output is horizontally connected to the HAFB group, which then combines the mask embedding vector and multi-scale image features to perform feature fusion and obtain fused features.

3. The infrared image target detection method based on joint learning of image and text masks according to claim 2, characterized in that: For each text-guided C3k2 module in the first TGC3k2 group, the high-level features are first downsampled to make their resolution the same, and then the two are concatenated. The concatenated image features After CBS processing, the image features are segmented into two parts according to the channel dimension. , Then image features The image features are obtained by processing the data in a cascaded C3k module. Then, the image features Together with the text embedding vector, the image is processed by the TG attention module to obtain image features. Finally, the image features and image features After performing a splicing operation and then processing it through CBS, the output features are obtained. ; For each text-guided C3k2 module in the second TGC3k2 group, the low-level features need to be upsampled to make the resolution of the current layer features the same, then concatenated, and the updated text embedding vector is sent to the TG attention module for processing. The remaining operations are the same as those in the text-guided C3k2 module in the first TGC3k2 group.

4. The infrared image target detection method based on joint learning of image and text masks according to claim 3, characterized in that: The text enhancement attention module first guides the output features of the C3k2 module to each text in the first TGC3k2 group. The text embedding vector is pooled separately with the mask embedding vector, and then the pooled features are concatenated. Finally, the text embedding vector is used as the query vector, and the concatenated features are used as the key vector and value vector. The text embedding vector is updated using a cross-attention mechanism.

5. The infrared image target detection method based on joint learning of image and text masks according to claim 3, characterized in that: The hierarchical attention fusion module adopts a symmetrical parallel structure design, including two sets of local branches and global branches, which are symmetrically distributed relative to the convolution branches. The first set of local and global branches takes the current layer features from the multi-scale image features as input, and processes them through LocalGlobalAttention attention modules with different patch parameters to obtain local features. and global features Then, the two are concatenated and output. The second set of local and global branches uses the output features of the current layer of the second set of TGC3k2 as input, and processes them through LocalGlobalAttention modules with different patch parameters to obtain local features. and global features Then, the two are concatenated and output. The convolutional branch takes the mask embedding vector as input, first adds it to the current layer features in the multi-scale image features and the output features of the current layer in the second TGC3k2 group, then concatenates it with the concatenated outputs of the two local branches and the global branches, and finally outputs the fused features through convolution.

6. The infrared image target detection method based on joint learning of image and text masks according to claim 2, characterized in that: The multi-scale image features are set as image features P2-P5. Both the first and second TGC3k2 groups have three text-guided C3k2 modules: a top-level module, a middle-level module, and a bottom-level module. The HAFB group has three hierarchical attention fusion modules. The image features P2-P4 are respectively connected to the input terminals of the three text-guided C3k2 modules of the first TGC3k2 group, the output terminals of the three text-guided C3k2 modules of the second TGC3k2 group are respectively connected to the input terminals of the three hierarchical attention fusion modules, the image features P3-P5 are also respectively connected to the input terminals of the three hierarchical attention fusion modules, and the output terminals of the three hierarchical attention fusion modules are all connected to the detection head; The image feature P5 is simultaneously connected to the top-level modules of the first TGC3k2 group and the second TGC3k2 group. The middle-level and bottom-level modules of the second TGC3k2 group are respectively connected to the top-level and middle-level modules of the first TGC3k2 group. The bottom-level module of the first TGC3k2 group is simultaneously connected to the bottom-level module and the detection head of the second TGC3k2 group.

7. The infrared image target detection method based on joint learning of image and text masks according to claim 1, characterized in that: The masking branch first processes the infrared image using the Segment Anything Model 3 (SAM3) to generate a target mask image, and then uses a mask encoder to process the target mask image and multi-scale image features to obtain a mask embedding vector. The text branch uses the Generative Pretraining Transformer (GPT) natural language model to process the infrared image, generating a text description containing target location, quantity, and background information, and then uses the CLIP text encoder to construct a text embedding vector.

8. The infrared image target detection method based on joint learning of image and text masks according to claim 1, characterized in that: Calculate the total loss function using the following formula. , in, This represents the predicted bounding box brightness value. This represents the brightness value of the actual bounding box. Let B represent the pixel-level difference, B represent the set of adjusted bounding boxes, and N represent the number of pixels in each bounding box. Indicates the distribution focus loss. and These represent IoU loss respectively. and brightness difference loss The weighting coefficients, and the sum of the two is 1. This represents the target-text contrast loss function. This represents the target-mask contrast loss function.

9. A computing storage medium for storing computer programs, characterized in that: When the computer program is executed by the processor, it causes the electronic device to perform the infrared image target detection method based on image-text mask joint learning as described in any one of claims 1-8.

10. An electronic device comprising a memory, a processor, and a computer program stored within the memory, characterized in that: When the processor executes the computer program, it causes the electronic device to perform the infrared image target detection method based on image-text mask joint learning as described in any one of claims 1-8.

Images