Infrared ship detection method based on improved RT-DETR algorithm

Abstract

The invention discloses an infrared ship detection method based on an improved RT-DETR algorithm, and aims to improve the detection precision and efficiency of a ship target in an infrared image. By introducing a multi-layer dynamic feature conversion module (MDST), the diversity of feature extraction is enhanced; effective fusion of multi-scale features is realized by using a high-level screening feature fusion pyramid network (HS-FPN); an original RT-DETR backbone network is replaced by a PPA and C2f structure, so that the problem of information loss in a down-sampling process is solved; and the CA module is used for enhancing the feature selection and fusion effect. Experimental results show that the provided method significantly improves the detection performance on an infrared marine ship data set, is suitable for resource-constrained equipment such as unmanned aerial vehicles and smart phones, and has a wide application prospect.

Description

Technical Field

[0001] The invention relates to water vessel detection, and in particular to an infrared vessel detection method based on an improved RT-DETR algorithm. Background Art

[0002] Infrared imaging occupies an important position in remote sensing technology due to its unique imaging capabilities. Compared with visible light imaging technology, infrared imaging can obtain images in real time under day and night conditions. By detecting the infrared radiation emitted by objects, infrared images can effectively penetrate common environmental obstacles such as smoke and mist to achieve all-weather imaging. With the continuous development of infrared imaging technology, image resolution and sensitivity have been significantly improved, and it has important application value in environmental monitoring, agricultural fields, night surveillance and target detection. At the same time, the all-weather and multi-angle imaging capabilities of infrared imaging make it also have important applications in maritime traffic monitoring and ship identification, effectively improving the level of maritime safety management. Although infrared imaging technology performs well in marine image acquisition, it still faces many difficulties in target detection. First, the marine environment and ship distribution are complex, and the image often contains multiple ships and complex backgrounds, with many interference factors, which increases the difficulty of identification. Secondly, when detecting at a long distance, small ships only occupy a small number of pixels in the infrared image, and sometimes present patchy or point-like features. In addition, the infrared reflection characteristics at different angles vary greatly, and it is difficult for the model to accurately capture and learn directional information. Finally, environmental noise such as sea surface thermal radiation can also interfere with infrared image detection, increasing the probability of false detection and missed detection. To solve these problems, more advanced image processing and target detection algorithms need to be developed to improve the accuracy and reliability of infrared ship detection.

[0003] In recent years, the Transformer architecture has performed significantly in optical image recognition. In 2020, the Facebook AI research team proposed the Detection Transformer (DETR), which achieved end-to-end object detection. In 2023, Lu Wenyu proposed RT-DETR, which surpassed similar detectors in both speed and accuracy. RT-DETR eliminates the region proposal network and non-maximum suppression (NMS) steps in traditional object detectors, and achieves higher prediction accuracy and inference speed through the global feature learning ability of the Transformer. However, RT-DETR has low performance when dealing with small targets. Subsequent studies have introduced conditional decoder attention mechanisms and deformable attention to improve the performance of small target detection. The C2f (Cross Stage Partial Network) structure is designed to achieve efficient feature extraction and fusion. It divides the input feature map into multiple groups through cross-stage partial connections and grouped convolution operations, and each group performs convolution operations separately to reduce the number of parameters and computational requirements, while improving the robustness and generalization of the network. The C2f structure can effectively process feature information of different scales and levels and improve the feature expression ability of the model. The PPA module extracts image features from different angles and scales by introducing multiple parallel branches in the feature extraction process. It combines local and global feature extraction strategies, and measures the mutual relationship and importance between different patches by calculating the attention matrix between non-overlapping patches, ensuring that key information is retained through multiple downsampling steps. The PPA module enhances the model's ability to understand complex image scenes and improves the accuracy of small target detection. The CA module calculates channel weights through global average pooling and maximum pooling to enhance the attention of important features. In the feature selection process, the CA module combines the two to extract and retain representative information while reducing information loss. Through the channel attention mechanism, the model can have a more comprehensive understanding of the entire infrared scene and improve the accuracy of ship detection. The SFF module aims to solve the problem of scale differences in infrared images and enhance the feature expression ability of the model through multi-scale feature fusion. It fuses high-level and low-level information in feature maps of different scales through a selective feature fusion mechanism to form a feature map with rich semantic information. The SFF module can better capture the subtle features of infrared ships and improve the accuracy and reliability of target detection. NMS technology is used in target detection to filter out redundant detection boxes to retain the boxes with the highest confidence. NMS calculates the overlap between candidate boxes and removes boxes with high overlap according to a set threshold, thereby reducing the probability of false detection and missed detection. NMS is an essential step in many object detection algorithms, but it may cause some target information to be lost when processing complex scenes.

[0004] The invention with patent number CN202410539689.9 proposes a method for infrared ship target detection and tracking based on deep learning. It collects infrared ship video images by operating a civilian drone for aerial photography, detects ship targets in infrared ship video images using the PP-YOLOE algorithm, and continuously tracks the detected ship targets using the ByteTrack algorithm. This method uses the PP-YOLOE algorithm to extract ship target features in infrared remote sensing videos, and combines the ByteTrack algorithm for target tracking. However, this method still faces certain challenges when dealing with complex backgrounds and small targets, and the detection accuracy and real-time performance need to be further improved.

[0005] The invention with patent number CN202410427419.9 proposes an infrared ship target detection method based on task alignment learning. By operating civilian drones to collect infrared ship video images in different locations and scenes, the YOLOX algorithm is improved to build an infrared ship target detection model, and the model is trained based on infrared ship video images. This method is superior to other similar methods in detection accuracy and real-time performance, but there is still room for improvement when dealing with multi-scale and complex backgrounds.

[0006] The invention with patent number CN202110944381.9 proposes a ship classification method and device based on deep learning fusion of visible light images and infrared images, designs a two-stream symmetric convolutional neural network that can obtain more global information of ship images, extracts deep features of visible light images and infrared images, and obtains ship classification results through linear weighted decision fusion. This method effectively utilizes the complementary information of multimodal images for fusion classification, but it still has certain limitations in complex scenes and multi-scale target detection.

[0007] Current infrared ship detection methods still face certain challenges in dealing with complex backgrounds, multi-scale targets, and small target detection. Specific issues include the following:

[0008] Limited detection accuracy: Existing methods have low detection accuracy when processing infrared images with multiple scales and complex backgrounds, and are prone to false detection and missed detection.

[0009] Insufficient computational efficiency: Some deep learning algorithms have high computational complexity and poor real-time performance, making them unsuitable for resource-constrained devices.

[0010] Small target detection is difficult: Small targets only occupy a small number of pixels in infrared images, and existing methods have difficulty in accurately capturing and detecting small targets.

[0011] Insufficient feature expression ability: During multiple downsampling processes, important feature information is easily lost, resulting in decreased detection accuracy. Summary of the invention

[0012] In view of the above problems, the present invention proposes an infrared ship detection method based on an improved RT-DETR algorithm. By introducing a multi-layer dynamic feature conversion module (MDST), a multi-branch feature extraction strategy (C2f-PPA) and a high-level feature fusion module (SFF), the accuracy and reliability of infrared ship detection are significantly improved while maintaining computational efficiency. The method specifically includes the following steps:

[0013] S1: Input infrared image containing ship object;

[0014] S2: Ship object detection is performed on infrared images by improving the RT-DETR algorithm module, and the ship object is output, including the main part, feature selection part, multi-scale feature fusion part, and decoding and detection part. The specific components are as follows:

[0015] T1: The main part uses the PPA module based on the C2f structure and performs multi-branch feature extraction through multiple downsampling operations;

[0016] T2: The feature selection part uses the CA module to perform weighted processing on the extracted features, calculates channel weights through global average pooling and maximum pooling, and selects features related to the task; the weighted features are processed by 1×1 convolution to unify the number of channels;

[0017] T3: The multi-scale feature fusion part uses the HS-FPN module to achieve the fusion of feature maps of different scales to form a feature map with rich semantic information. The SFF module and MDST module are used to further enhance the stability and accuracy of the model.

[0018] T4: Decoding and detection part: The fused feature map is input into the decoder and detection head to perform target detection and output the ship detection results in the infrared image.

[0019] As a technical preferred solution of the present invention, the multi-branch feature extraction path extracts image features from different angles and scales, including a local branch, a global branch and a serial convolution branch, and specifically includes the following steps:

[0020] Given an input feature tensor F∈R H′×W ' ×C , first adjust by point-wise convolution to get F'∈R H′×W′×C′ ; Where F is the input feature tensor, H' is the feature tensor height, W' is the feature tensor width, C is the number of feature tensor channels, and R is the real number domain;

[0021] Calculate the local branches F separately local ∈R H′×W′×C′ , global branch F global ∈R H′×W′×C′and serial convolution F conv ∈R H′×W′×C′ ; Among them, F local Represents the feature tensor extracted by the local branch, F global represents the feature tensor extracted by the global branch, F conv represents the feature tensor extracted by the serial convolution branch, and C' represents the number of channels of the adjusted feature tensor; the feature extraction process of the local branch includes using a smaller convolution kernel to perform a convolution operation to extract detail features; the feature extraction process of the global branch includes using a larger convolution kernel to perform a convolution operation to extract global features; the serial convolution branch performs continuous convolution processing through a series of convolution operations to capture continuous feature information;

[0022] The feature tensors extracted by local branch, global branch and serial convolution are added together to obtain in, is the final feature map.

[0023] As a technical preferred solution of the present invention, the processing of the local branch and the global branch includes the following steps:

[0024] G1: Definition of the image patch size. The parameter p is used to define the size of the image patch. The input feature tensor F is divided into multiple non-overlapping regions, and the size of each region is determined by p.

[0025] G2: Weighted processing of feature tensors, including the following steps:

[0026] G1-1: weight the features in the feature tensor to select the features relevant to the task;

[0027] G1-2: The dimension after dividing the feature tensor into non-overlapping patches is represented as:

[0028]

[0029] Where d represents the dimension after patching, H' and W' represent the height and width of the feature tensor respectively;

[0030] G1-3: Calculate the weighted result. The specific calculation formula is:

[0031]

[0032] Among them, each t i ∈R d represents the i-th output token, c' represents the number of output tokens;

[0033] G3: Feature selection operation, including the following steps:

[0034] G3-1: For each mark t iPerform feature selection operation. The calculation formula for the selection operation is:

[0035]

[0036] Among them, ξ∈R C′ and P∈R C′×C′ is a task-specific parameter, sim(·,·) is a function for calculating cosine similarity, ranging from [0,1];

[0037] G3-2: Task embedding through ξ, each tag t i They are re-weighted according to their relevance to the task embedding, effectively simulating token selection;

[0038] G4: Linear transformation and interpolation operation of parameter matrix. The specific steps are as follows:

[0039] G4-1: linearly transform the parameter matrix P and select relevant channels for each marker;

[0040] G4-2: Perform reshaping and interpolation operations to calculate:

[0041] F local ∈R H′×W′×C′

[0042] F global ∈R H′×W′×C′

[0043] G5: Sequential convolution operation. The specific steps are as follows:

[0044] Replacing the 7×7, 5×5, and 3×3 convolutional layers with a sequential convolution consisting of three 3×3 convolutional layers yields:

[0045] F conv1 ∈R H′×W′×C′

[0046] F conv2 ∈R H′×W′×C′

[0047] F conv3 ∈R H′×W′×C′

[0048] Among them, F conv1 、F conv2 、F conv3 Represents three convolution output results, which are added together to form the final output F of the sequence convolution conv , F conv ∈R H'×W'×C' .

[0049] As a technical preferred solution of the present invention, after extracting the final feature map, adaptively enhancing the features using the attention mechanism can improve the performance of the model, which specifically includes the following steps:

[0050] First, the feature map After the one-dimensional channel attention map M C ∈R 1×1×C′ Calculate the importance weight of each channel;

[0051] Then, after the two-dimensional spatial attention map M S ∈R H′×W′×1 Calculate the importance weight of each position in the feature map. The specific calculation formula is as follows:

[0052]

[0053] in, represents element-wise multiplication, F C ∈R H×W×C′ and F S ∈R H×W×C′ represents the characteristics after channel and space selection, δ(·) and ReLU and batch normalization BN, respectively, F”∈R H×W×C′ is the final output of the PPA.

[0054] As a technical preferred solution of the present invention, the HS-FPN module in T3 captures and fuses infrared ship features of different scales, thereby coping with the multi-scale problem in target detection, and is specifically composed of a feature selection module and a feature fusion module.

[0055] As a technical preferred solution of the present invention, the feature selection module is divided into CA and DM, which specifically includes the following steps:

[0056] Y1: The CA module calculates channel weights through global average pooling and maximum pooling to enhance the attention of important features. The specific steps are as follows:

[0057] Y1-1: input feature map f in ∈R C×H×W , where C represents the number of channels, H represents the height of the feature map, and W represents the width of the feature map;

[0058] Y1-2: Calculate feature map f through global average pooling and maximum pooling in The channel weights are obtained to obtain two feature maps f avg and f max ;

[0059] Y1-3: f avg and f maxThrough the fully connected layer and processed with the Sigmoid activation function, the weight W of each channel is obtained;

[0060] Y1-4: put f in The weighted feature map f is obtained by feature fusion calculation with W weighted ;

[0061] Y2: Use the dimension matching module to unify the feature map dimensions. Before feature fusion, since feature maps of different scales have different numbers of channels, 1×1 convolution is used to reduce the number of channels of the feature map to 256. The specific calculation formula is as follows:

[0062] f DM =Conv1x1(f weighted )

[0063] Among them, Conv1x1 represents a 1×1 convolution operation, f DM Represents the feature map after dimension matching.

[0064] As a technical preferred solution of the present invention, the feature fusion module includes the following steps:

[0065] Input high-level feature maps and low-level feature maps: high-level feature map f high ∈RC ×H×W , where C represents the number of channels, H represents the height of the feature map, and W represents the width of the feature map; low-level feature map Where H1 represents the height of the low-level feature map, and W1 represents the width of the low-level feature map;

[0066] Advanced feature size matching: The advanced feature f high The deconvolution operation with a stride of 2 and a convolution kernel of 3×3 is used to enlarge its size to The semantic information is retained while increasing the resolution of the feature map; high-level features are upsampled or downsampled using bilinear interpolation to unify the dimensions.

[0067] Feature fusion: The CA module is used to generate attention weights using the rich semantic information provided by high-level features and filter the basic information in low-level features. The specific feature fusion calculation formula is:

[0068] f att =BL(T-Conv(f high ))

[0069] f att =f low *CA(f att )+f att

[0070] Through feature fusion, we get Among them, CA represents the channel attention module, f att is the processed high-level feature, f low is the low-level feature map, f high is the low-level feature map, f out is the fused output feature map, BL is the bilinear interpolation calculation, and T-Conv is the deconvolution calculation.

[0071] As a technical preferred solution of the present invention, the MDST module in T3 integrates group convolution, channel shuffling, visual converter and depth-separable convolution technology, and specifically includes the following features:

[0072] Group convolution divides the input channels into multiple groups, and each group performs a separate convolution operation;

[0073] Channel shuffling technology, by exchanging features between groups, enables the features of different groups to merge and complement each other;

[0074] The visual converter splits the input features in a 3:1 ratio, and performs group convolution and channel shuffling operations on some of the features to reduce computing requirements and maintain feature diversity. It replaces the fully connected linear layer with convolution operations to reduce computing requirements and improve model performance.

[0075] Depthwise separable convolution uses depthwise convolution to extract spatial features and extracts channel features through point-by-point convolution, reducing the amount of computation and improving computational efficiency. Three convolutional layers are used to complete the tasks of the fully connected linear layer, further reducing computational requirements and improving model performance.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] The present invention effectively improves the diversity and richness of feature extraction by introducing a multi-layer dynamic feature conversion module (Multi-layer Dynamic Shuffle Transformer, MDST), combining channel shuffle and group convolution. This module can significantly enhance the model's detection accuracy for small targets and ship targets in complex backgrounds.

[0078] The present invention integrates the Vision Transformer technology in the MDST module, which significantly reduces the computational complexity and memory requirements of the model while maintaining high accuracy. By introducing Depthwise Separable Convolutions, the computational efficiency and adaptability of the model are further improved, making it more suitable for deployment on resource-constrained devices.

[0079] This paper proposes a high-level screening-feature fusion pyramid network (HS-FPN), which realizes the effective fusion of features of different scales. HS-FPN integrates high-level and low-level feature information through a selective feature fusion mechanism, significantly enhancing the model's ability to detect targets of different scales.

[0080] The present invention combines the Parallelized Patch-aware Attention (PPA) module with the C2f structure, replaces the backbone network of the original RT-DETR, and adopts a multi-branch feature extraction strategy to ensure that key information is retained in multiple downsampling steps, thereby improving the detection accuracy of the lightweight model.

[0081] The present invention adopts the channel attention module (CA) in the feature selection process, calculates the channel weights through global average pooling and maximum pooling, and enhances the attention of important features. In the feature fusion module, the CA module is used to generate attention weights using the rich semantic information provided by high-level features, filter the basic information in low-level features, improve the effect of feature fusion, and finally output high-quality detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 A method flow chart of an infrared ship detection method based on an improved RT-DETR algorithm provided by the present invention;

[0083] Figure 2 A network structure diagram of an improved RT-DETR algorithm provided in an embodiment of the present invention;

[0084] Figure 3 A network structure diagram of C2f-PPA providing an embodiment of the present invention;

[0085] Figure 4 A network structure diagram of a PPA providing an embodiment of the present invention;

[0086] Figure 5 A network structure diagram of SFF and CA providing an embodiment of the present invention;

[0087] Figure 6 A network structure diagram of MDST providing an embodiment of the present invention;

[0088] Figure 7 A size distribution diagram of a ship tag frame according to an embodiment of the present invention is provided;

[0089] Figure 8The present invention provides PR curves of RT-DETR and MAFF-DETR of the embodiments;

[0090] Fig. 9 The present invention provides a ship detection effect diagram of an embodiment. DETAILED DESCRIPTION

[0091] The present invention is further described below in conjunction with the accompanying drawings and examples. However, the present invention can be implemented in many different ways and should not be construed as being limited to the embodiments shown; on the contrary, these embodiments provide those skilled in the art with implementation methods that meet applicable legal requirements.

[0092] Example 1: Figure 1 As shown, this embodiment provides an infrared ship detection method based on an improved RT-DETR algorithm, wherein the improved RT-DETR algorithm is specifically a MAFF-DETR network, and the method includes the following steps:

[0093] S1: Input infrared image containing ship object;

[0094] S2: Ship object detection is performed on infrared images by improving the RT-DETR algorithm module, and the ship object is output, including the main part, feature selection part, multi-scale feature fusion part, and decoding and detection part. The specific components are as follows:

[0095] T1: Main part: Use the PPA module based on the C2f structure to perform multi-branch feature extraction through multiple downsampling operations;

[0096] T2: Feature selection part: The CA module is used to weight the extracted features, and the channel weights are calculated through global average pooling and maximum pooling to select the features related to the task; the weighted features are processed by 1×1 convolution to unify the number of channels;

[0097] T3: Multi-scale feature fusion part: Use the HS-FPN module to achieve the fusion of feature maps of different scales to form a feature map with rich semantic information, and further enhance the stability and accuracy of the model through the SFF module and MDST module;

[0098] T4: Decoding and detection part: The fused feature map is input into the decoder and detection head to perform target detection and output the ship detection results in the infrared image.

[0099] like Figure 2As shown, the improved RT-DETR algorithm described in the present invention is specifically the network structure of the MAFF-DETR network. In the backbone part, the present invention uses a PPA attention module based on the C2f structure to solve the problem of losing key information of small infrared objects in multiple downsamplings. Due to the large differences in shooting distances of infrared images, in the neck part, the present invention adopts the HS-FPN network structure to achieve multi-scale feature fusion. The MDST module proposed in the present invention cleverly combines channel shuffling technology and group convolution, achieving a good balance between model complexity and performance. This design not only improves computational efficiency, but also retains powerful information extraction capabilities. In addition, the MDST module further enhances the stability and accuracy of the model by introducing skip connections and depthwiseseparable convolutions.

[0100] like Figure 3 As shown, the present invention replaces the traditional convolution operation in the backbone network by combining PPA with the C2f module, aiming to solve the problem that in the infrared small object detection task, small objects are prone to lose key information in multiple downsampling operations.

[0101] The core idea of ​​multi-branch feature extraction is to extract image features from different angles and scales through multiple parallel feature extraction paths, such as Figure 4 As shown in Figure 2. These paths can use different convolution kernel sizes, layers or structures. The multi-branch strategy helps capture the diversity and complexity of object images and improves the accuracy of small object detection. Through parallel processing, the model can efficiently fuse features from different paths to form a richer and more comprehensive image representation. Specifically, this strategy involves three parallel branches: local branch, global branch and serial convolution branch. Given an input feature tensor F∈R H′×W′×C , first adjust by point-wise convolution to get F'∈R H′×W′×C′ Then, through these three branches, F is calculated respectively. local ∈R H′×W′×C′ 、F global ∈R H′×W′×C′ and F conv ∈R H′×W′×C′ Finally, we add these three results together to get

[0102] The main difference between the local branch and the global branch is that they process different sizes of image patches. The local branch focuses on detail features, while the global branch focuses on global features. This distinction is achieved through the parameter p, which defines the size of the patch. p is set by aggregating and shifting non-overlapping patches in the image space dimension, that is, the image is divided into multiple non-overlapping regions, each of which is determined by p. In order to effectively extract and interact local and global features, the attention matrix between non-overlapping patches is also calculated to measure the relationship and importance between different patches. This helps the model capture the details and global information of the image at different scales, improving the quality and efficiency of feature extraction.

[0103] In the feature map after preliminary processing, the features need to be weighted to select the features relevant to the task. The dimension after dividing the feature map into non-overlapping patches is represented as The weighted result is expressed as Each t i ∈R d represents the i-th output token. For each token t i Perform feature selection operation, and the output result is where ξ∈R C′ and P∈R C′×C′ is a task-specific parameter, sim(·,·) is a function that computes cosine similarity in the range [0,1]. ξ plays the role of task embedding, specifying which tags are relevant to the task. Each tag t i Each of them is reweighted according to its relevance to the task embedding, effectively simulating token selection. Subsequently, the parameter matrix p is linearly transformed to select the relevant channels for each token. This is followed by reshaping and interpolation operations to produce F local ∈R H′×W′×C′ and F global ∈R H′×W′×C′ Finally, a sequential convolution consisting of three 3×3 convolutional layers is used to replace the traditional 7×7, 5×5, and 3×3 convolutional layers. conv1 ∈R H′×W′×C′ , F conv2 ∈R H′×W′×C′ and F conv3 ∈R H′×W′×C′ , add the three convolution output results to get the final output F of the sequence convolution conv ∈R H′×W′×C′ .

[0104] After obtaining the feature map through multi-branch feature extraction, adaptively enhancing the features using the attention mechanism can improve the performance of the model. The attention module consists of efficient channel attention and spatial attention. First, the feature map After the one-dimensional channel attention map M C ∈R1×1×C′ Calculate the importance weight of each channel and then pass it through the two-dimensional spatial attention map M S ∈R H′×W′×1 Calculate the importance weight of each position in the feature map. This process can be summarized as follows:

[0105]

[0106] in, represents element-wise multiplication, F C ∈R H×W×C′ and F S ∈R H×W×C′ represents the characteristics after channel and space selection, δ(·) and Represents Rectified Linear Unit (ReLU) and Batch Normalization (BN), respectively, F”∈R H×W×C′ is the final output of the PPA.

[0107] In order to solve the multi-scale challenges inherent in the infrared ship dataset, the present invention introduces Hierarchical Scale-based Feature Pyramid Network (HS-FPN) to achieve multi-scale feature fusion. HS-FPN aims to capture and fuse infrared ship features of different scales, thereby coping with the multi-scale problem in target detection. Its structure mainly includes two key components: a feature selection module and a feature fusion module. Initially, feature maps of different scales are screened and processed in the feature selection module to extract and enhance useful information. Subsequently, a selective feature fusion mechanism is used to synergistically integrate high-level and low-level information in feature maps of different scales. This fusion forms a feature map with rich semantic information, which can better detect the subtle features of targets in the infrared ship dataset, thereby significantly enhancing the detection capability of the model.

[0108] The feature selection module consists of two parts: channel attention (CA) and dimension matching (DM). The CA module calculates channel weights through global average pooling and maximum pooling to enhance the attention of important features, such as Figure 5 As shown. This module initially processes the input feature map f in ∈R C×H×W, where C represents the number of channels, H represents the height of the feature map, and W represents the width of the feature map. The processed feature map uses the Sigmoid activation function to obtain the weight of each channel, and then multiplies it with the corresponding feature to generate a weighted feature map. Maximum pooling extracts the significant features of each channel to ensure that important features are not ignored; average pooling obtains the average information of each channel to reduce information loss. The CA module combines the two to comprehensively extract and retain representative information while reducing information loss. Before feature fusion, since feature maps of different scales have different numbers of channels, direct fusion will lead to size mismatch problems. To this end, the DM module is used to uniformly reduce the number of channels of the feature map to 256 by 1×1 convolution.

[0109] The feature fusion module aims to fuse the multi-scale feature maps generated from the backbone network to achieve accurate object detection. In the multi-scale feature maps, high-level features have rich semantic information but relatively rough target positioning, while low-level features contain accurate target location information but less semantic information. Figure 5 As shown in Figure 2, in order to fuse high-level features with low-level features, they first need to match their sizes. Given an input high-level feature f high ∈R C×H×W and an input low-level feature The high-level features are enlarged to a size of 2 using a transposed convolution with a stride of 2 and a convolution kernel of 3×3. While increasing the resolution of the feature map, the semantic information is retained. Then, the high-level features are upsampled or downsampled using Bilinear Interpolation to unify the dimensions. Then, the CA module is used to generate attention weights using the rich semantic information provided by high-level features and filter the basic information in low-level features, thereby achieving efficient feature fusion and finally obtaining The following is the fusion process of feature selection:

[0110] f att =BL(T-Conv(f high )),

[0111] f att =f low *CA(f att )+f att ,

[0112] like Figure 6As shown in Figure 1, group convolution aims to reduce the number of parameters and computational requirements of the model while improving the robustness and generalization ability of the network. It divides the input channels into multiple groups, and each group performs a convolution operation separately. This method brings the following benefits: by grouping the input channels, group convolution significantly reduces the number of convolution kernels; the reduction in the number of parameters helps prevent model overfitting and improves the generalization ability of the model; the group convolution operation performs partial independent processing of the input features, making the model more robust in the face of noise and interference. In addition, the present invention draws on the channel shuffling technology of ShuffleNetV2 to promote the exchange of feature information between groups, so that the features of different groups can be integrated and complemented with each other, and the overall feature expression ability is enhanced. Although the number of parameters is reduced, the diversity and richness of the features are maintained through the channel shuffling operation, avoiding feature redundancy and information loss that may be caused by group convolution. The combined use of group convolution and the channel shuffling technology of ShuffleNetV2 not only reduces the number of parameters and computational requirements of the model, but also maintains the diversity and richness of the features.

[0113] The MDST module innovatively combines visual converters to further improve the computational efficiency and performance of the model. In MDST, the input features are split in a 3:1 ratio, and a portion of them undergo group convolution and channel shuffling, which reduces computational requirements while maintaining feature diversity. Considering that convolution operations are less computationally intensive than fully connected layers and are more suitable for processing large-scale image data and local features, the present invention replaces the fully connected linear layer with a convolution operation. This design not only reduces computational requirements, but also better adapts to the characteristics of convolutional neural networks, helping to improve model performance. The present invention uses three convolutions to complete the tasks of the fully connected linear layer, in which DSConv extracts spatial features through deep convolution and extracts channel features through point-by-point convolution, which can reduce the amount of computation while improving computational efficiency.

[0114] Example 2: The dataset used in this example comes from the InfiRay infrared open platform. More than 8,000 infrared images were collected in coastal ports and docks, with resolutions of 384×288, 640×512, and 1280×1024. The size distribution of the ship tag frame is as follows: Figure 7 As shown in the figure, it contains ships of various sizes, which tests the multi-scale detection ability of the model. The ship categories in the dataset include liner, bulk carrier, warship, sailboat, canoe, container ship and fishing boat. Among them, fishing boat, sailboat and canoe account for the majority, reflecting the real ocean scene. The image data is divided into training set, validation set and test set in a ratio of about 7:1:2, specifically 5881 training images, 840 validation images and 1681 test images.

[0115] The operating system used in this embodiment is ubuntu20, equipped with Intel (R) Xeon (R) CPU E5-2680v4 CPU and RTX 3090 (24G) GPU. The experiment was performed using the PyTorch framework (version 2.0.0), and the parameter settings were based on the default parameters of RT-DETR. The batch size used in the present invention is 16. The size of the input image is adjusted and randomly cropped to a size of 640×640. The initial learning rate is 0.0001 and the weight decay is 0.0001.

[0116] In order to fully demonstrate the performance of the proposed MAFF-DETR, we use precision, recall, mAP50 and mAP50:95 as evaluation metrics.

[0117] Precision refers to the proportion of samples predicted as positive by the model that are actually positive, which measures the accuracy of the model in positive prediction. Recall refers to the proportion of samples that are actually positive that are correctly predicted as positive by the model, which measures the comprehensiveness of the model in identifying positive samples. The formulas for precision and recall are as follows:

[0118]

[0119] Among them, "true positive" refers to the number of samples correctly identified as the positive category, "false positive" refers to the number of samples incorrectly identified as the positive category, and "false negative" refers to the number of samples incorrectly identified as the negative category.

[0120] In object detection, mean average precision (mAP) measures the model effect by evaluating the precision-recall curve (PR curve). mAP50 refers to the mean average precision when the IoU threshold is 0.5, and mAP50:95 refers to the mean average precision when the IoU threshold ranges from 0.5 to 0.95 with a step size of 0.05. GFLOPs refers to the number of floating-point operations performed per second, which measures the computational performance of the model. Parameters refer to the weights and biases of the model, which determine the computational complexity.

[0121] The neck part of the present invention selects Channel Attention instead of other attention mechanisms because infrared thermal images are global. Channel Attention can better capture global context information and more comprehensively understand the entire infrared scene, thereby improving the accuracy of ship detection. Channel Attention integrates bottleneck transformation and reduces the redundancy of global context features. Although it slightly increases the computational complexity, it improves the detection effect.

[0122] Table 1: Experimental results of different improvements

[0123]

[0124] In order to prove the effectiveness of the feature fusion module designed by the present invention, multiple attention mechanisms are used for fusion. Under the same parameter settings and without pre-trained weights, considering model parameters, GFLOPs and mAP, Channel Attention, Efficient Local Attention, Coordinate Attention and Context AnchorAttention are used in collaboration with C2f-PPA for comparative experiments. The attention collaboration method proposed in the present invention performs well in average detection accuracy, and the parameters and computational load are on par with other algorithms. Experiments show that Channel Attention more comprehensively extracts and retains the most representative information in each channel. The results are shown in Table 1.

[0125] Example 3: In order to evaluate the effectiveness of the MAFF-DETR model, this example conducted an ablation experiment and compared it with the original RT-DETR algorithm (S0). The results of the ablation study are shown in Table 2, which shows six schemes (S0-S5), each of which represents a combination of different improved strategies. S0 does not apply any strategy, while S5 implements all five strategies.

[0126] Table 2: MAFF-DETR ablation test results

[0127]

[0128] In this embodiment, the present invention uses sequential stacking modules for comparison. First, only the original RT-DETR model is used for testing, and then C2f-PPA, HS-FPN and MDST are added in sequence. The results are shown in Table 2. After adding C2f-PPA, mAP50:95 is increased by 0.9%, the number of parameters is reduced by 1.9M, and the amount of calculation is increased by 3.2GFLOPs. After adding C2f-PPA and HS-FPN, the mAP is improved by reducing the number of parameters by 1.7M and the amount of calculation by 3.6GFLOPs compared with S1. After continuing to add the MDST module, compared with S4, the mAP is increased by 0.5% with a reduction of 0.7M parameters and a reduction of 5.4GFLOPs in calculation. Compared with the original RT-DETR-r18 model, the final model has a 4.3M reduction in parameters, a 5.8GFLOPs reduction in calculation, and a 1.7% increase in mAP. The results show that the model reduces the computational cost while improving the accuracy, and is suitable for infrared ship detection.

[0129] The PR curve shows the trade-off between the precision and recall of the classification model at different thresholds, and evaluates the performance of the model on an imbalanced dataset and at different thresholds. The closer the PR curve is to the upper right corner, the higher the precision and recall of the model, and the better the overall performance. Figure 8The PR curves of RT-DETR and MAFF-DETR are shown, indicating that MAFF-DETR performs well between precision and recall with a higher true positive rate.

[0130] Table 3: Comparison results of average precision of each category.

[0131]

[0132]

[0133] The module proposed in this paper has higher accuracy and good performance in multi-scale and small target detection. Bulk carriers, container ships and warships have higher detection accuracy because they are generally small in number and large in size. Fishing boats, sailboats and canoes are numerous and contain many small targets, which are easily mixed with complex backgrounds, increasing the difficulty of detection. Therefore, the average accuracy of these three types of ships is lower, as shown in Table 3.

[0134] Example 4: This example compares the proposed MAFF-DETR with the YOLO series algorithms without pre-trained weights. Compared with YOLOv6m, MAFF-DETR shows better detection results with fewer parameters and computation. Compared with YOLOv9c, MAFF-DETR significantly reduces parameters and computational costs while ensuring accuracy. Compared with the latest YOLOv10, MAFF-DETR also shows excellent detection performance. The comparison results of the infrared ship dataset are shown in Table 4.

[0135] Table 4: Comparison results with YOLO series algorithms

[0136] Parameters(M) FLOPs(G) mAP50(%) mAP50:95(%) YOLO10m 16.5 63.5 0.912 0.645 YOLO9c 25.5 102.8 0.919 0.65 YOLO8m 25.9 78.9 0.913 0.651 YOLO6s 16.3 44 0.895 0.632 YOLO6m 52 161.2 0.906 0.64 YOLO6l 110.9 391.2 0.905 0.641 YOLO5m 25.1 64.2 0.914 0.65 YOLO5l 53.2 135 0.916 0.651 Ours 15.6 51.2 0.935 0.65

[0137] Table 5: Comparison results with other algorithms

[0138]

[0139]

[0140] In order to verify the effectiveness of the proposed algorithm, it was compared with ATSS, Cascade-RCNN, DDQ-DETR, DINO, Faster-RCNN, GFL, RetinaNet and TOOD, all of which used ResNet50 as the backbone network. To ensure fair comparison, these algorithms were trained on the same dataset and under the same experimental conditions. The results are shown in Table 5. The mAP of the model of the present invention is 1.7% higher than that of the highest accuracy RT-DETR-R18 before the improvement. Comparison with other advanced algorithms shows that the model of the present invention achieves higher accuracy while significantly reducing the number of parameters and computational cost, demonstrating its lightweight characteristics and being suitable for deployment on equipment such as unmanned ships. The effectiveness of the model can be attributed to the efficient combination of multidimensional feature extraction and MDST feature enhancement of the HS-FPN module, as well as the reduction of computational requirements by jump connections, which helps to achieve high accuracy with minimal computation.

[0141] The MDST module performs well in processing multi-scale information, enhancing small object features through dynamic grouped convolution and feature shuffling, and outperforming advanced algorithms such as the YOLO series and DINO in infrared ship detection. It has advantages in accuracy and efficiency, especially when processing scenes with a lot of details and dynamic range. MDST also optimizes computing resources to maintain high performance even in resource-constrained environments.

[0142] In order to intuitively demonstrate the performance of MAFF-DETR, the present invention selects 6 common scenarios and compares the detection results of the original RT-DETR model and the present invention model. Fig. 9 As shown, the infrared image on the far left has fewer texture features and lower contrast between the target and the background. Fig. 9 The a in the figure shows that the RT-DETR model cannot detect multiple targets, while the model of the present invention can recognize subtle texture features and distinguish them. Fig. 9 The target area in b is smaller and has night reflections. Although all methods have a certain degree of incomplete recognition, MAFF-DETR can identify more ships and performs better in detecting small targets and dense targets at the sea-sky boundary. Fig. 9 Figure c shows that RT-DETR misdetects the shore as a ship, while the model of the present invention can accurately identify shore buildings and ships in the water. Fig. 9Figures d, e, and f show more complex scenes where ships easily blend into the environment and have serious occlusion problems. The model of the present invention still shows high accuracy, especially in identifying occluded sailboats and canoes at the edge of the picture. This difference may be because larger targets hinder the detection of smaller targets when adjacent targets are very close. The model of the present invention introduces C2f-PPA to extract features of different scales and levels, improves the accuracy of small object detection by capturing multi-scale features of objects, and can accurately identify objects of various sizes.

[0143] The following conclusions can be drawn from the above embodiments: According to the experimental results of the public infrared ship dataset, a series of ablation experiments prove that the backbone, neck and MDST modules of the present invention can effectively improve the detection performance. Compared with the original algorithm, the improved algorithm has an increase of 1.7% in mAP, a reduction of 4.3M parameters, and a reduction of 5.8GFLOPs in computation. This trade-off effectively balances the requirements of detection accuracy and model complexity, has made progress in the field of target detection on mobile devices, and has enhanced the practicality and application potential of the model in resource-constrained environments, becoming an important milestone in infrared ship detection.

[0144] The above embodiments only express the implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. An infrared ship detection method based on an improved RT-DETR algorithm, characterized in that: The method comprises the following steps: S1: Input infrared image containing ship object; S2: Ship object detection is performed on infrared images by improving the RT-DETR algorithm module, and the output includes ship object, trunk part, feature selection part, multi-scale feature fusion part, and decoding and detection part. It consists of the following modules: T1: Main part: Use the PPA module based on the C2f structure to perform multi-branch feature extraction through multiple downsampling operations; T2: Feature selection part: The CA module is used to weight the extracted features, and the channel weights are calculated through global average pooling and maximum pooling to select the features related to the task; the weighted features are processed by 1×1 convolution to unify the number of channels; T3: Multi-scale feature fusion part: Use the HS-FPN module to achieve the fusion of feature maps of different scales to form a feature map with rich semantic information, and further enhance the stability and accuracy of the model through the SFF module and MDST module; T4: Decoding and detection part: The fused feature map is input into the decoder and detection head to perform target detection and output the ship detection results in the infrared image.

2. The infrared ship detection method based on the improved RT-DETR algorithm according to claim 1, characterized in that: The multi-branch feature extraction in T1 extracts image features from different angles and scales, including local branches, global branches, and serial convolution branches. Specifically, it includes the following steps: Given an input feature tensor F∈R H×W×C , first adjust by point-by-point convolution to get F′∈R H′×W′×C′ ; Where F is the input feature tensor, F′ is the adjusted feature tensor, H′ is the feature tensor height, W′ is the feature tensor width, C is the number of feature tensor channels, and R is the real number domain; Calculate the local branches F separately local ∈R H′×W′×C′ , global branch F global ∈R H′×W′×C′ and serial convolution F conv ∈R H ′×W′×C′ ; Among them, F local Represents the feature tensor extracted by the local branch, F global represents the feature tensor extracted by the global branch, F conv represents the feature tensor extracted by the serial convolution branch, and C′ represents the number of channels of the adjusted feature tensor; the feature extraction process of the local branch includes using a smaller convolution kernel to perform a convolution operation to extract detail features; the feature extraction process of the global branch includes using a larger convolution kernel to perform a convolution operation to extract global features; the serial convolution branch performs continuous convolution processing through a series of convolution operations to capture continuous feature information; The feature tensors extracted by local branch, global branch and serial convolution are added together to obtain in, is the final feature map.

3. The infrared ship detection method based on the improved RT-DETR algorithm according to claim 2 is characterized in that: The processing of the local branch and the global branch comprises the following steps: G1: Definition of the image patch size. The parameter p is used to define the size of the image patch. The input feature tensor F is divided into multiple non-overlapping regions, and the size of each region is determined by p. G2: Weighted processing of feature tensors, including the following steps: G1-1: weight the features in the feature tensor to select the features relevant to the task; G1-2: The dimension after dividing the feature tensor into non-overlapping patches is represented as: Where d represents the dimension after patching, H′ and W′ represent the height and width of the feature tensor respectively; G1-3: Calculate the weighted result. The specific calculation formula is: Among them, each t i ∈R d represents the i-th output token, c′ represents the number of output tokens; G3: Feature selection operation, including the following steps: G3-1: For each mark t i Perform feature selection operation. The calculation formula for the selection operation is: Among them, ξ∈R C′ and P∈R C′×C′ is a task-specific parameter, sim(·,·) is a function for calculating cosine similarity, ranging from [0, 1]; G3-2: Task embedding through ξ, each tag t i They are re-weighted according to their relevance to the task embedding, effectively simulating token selection; G4: Linear transformation and interpolation operation of parameter matrix, including the following steps: G4-1: linearly transform the parameter matrix P and select relevant channels for each marker; G4-2: Perform reshaping and interpolation operations to calculate: F local ∈R H′×W′×C′ F global ∈R H′×W′×C′ G5: Sequential convolution operation, including the following steps: Replacing the 7×7, 5×5, and 3×3 convolutional layers with a sequential convolution consisting of three 3×3 convolutional layers yields: F conv1 ∈R H′×W′×C′ F conv2 ∈R H′×W′×C′ F conv3 ∈R H′×W′×C′ Among them, F conv1 、F conv2 、F conv3 Represents three convolution output results, which are added together to form the final output F of the sequence convolution conv , F conv ∈R H′×W′×C′ .

4. The infrared ship detection method based on the improved RT-DETR algorithm according to claim 3 is characterized in that: After the final feature map is extracted in step G5, the performance of the model can be improved by adaptively enhancing the features using the attention mechanism, which specifically includes the following steps: First, the feature map After the one-dimensional channel attention map M C ∈R 1×1×C′ Calculate the importance weight of each channel; Then, through the two-dimensional spatial attention map M S ∈R H′×W′×1 Calculate the importance weight of each position in the feature map. The specific calculation formula is as follows: in, represents element-wise multiplication, F C ∈R H×W×C′ and F S ∈R H×W×C′ represents the characteristics after channel and space selection, δ(·) and ReLU and batch normalization BN, respectively, F″∈R H×W×C′ is the final output of the PPA.

5. The infrared ship detection method based on the improved RT-DETR algorithm according to claim 1, characterized in that: The HS-FPN module described in T3 captures and fuses infrared ship features of different scales, thereby coping with the multi-scale problem in target detection. It is specifically composed of a feature selection module and a feature fusion module.

6. The infrared ship detection method based on the improved RT-DETR algorithm according to claim 5, characterized in that: The feature selection module is divided into CA and DM. The following steps are involved: Y1: The CA module calculates channel weights through global average pooling and maximum pooling to enhance the attention of important features. The specific steps are as follows; Y1-1: input feature map f in ∈R C×H×W , where C represents the number of channels, H represents the height of the feature map, and W represents the width of the feature map; Y1-2: Calculate feature map f through global average pooling and maximum pooling in The channel weights are obtained to obtain two feature maps f avg and f max ; Y1-3: f avg and f max Through the fully connected layer and processed with the Sigmoid activation function, the weight W of each channel is obtained; Y1-4: put f in The weighted feature map f is obtained by feature fusion calculation with W weighted ; Y2: Use the dimension matching module to unify the feature map dimensions. Before feature fusion, since feature maps of different scales have different numbers of channels, 1×1 convolution is used to reduce the number of channels of the feature map to 256. The specific calculation formula is as follows: f DM =Conv1x1(f weighted ) Among them, Conv1x1 represents a 1×1 convolution operation, f DM Represents the feature map after dimension matching.

7. The infrared ship detection method based on the improved RT-DETR algorithm according to claim 5 is characterized in that: The feature fusion module includes the following steps: Input high-level feature maps and low-level feature maps: high-level feature map f high ∈R C×H×W , where C represents the number of channels, H represents the height of the feature map, and W represents the width of the feature map; low-level feature map Where H1 represents the height of the low-level feature map, and W1 represents the width of the low-level feature map; Advanced feature size matching: The advanced feature f high The deconvolution operation with a stride of 2 and a convolution kernel of 3×3 is used to enlarge its size to The semantic information is retained while increasing the resolution of the feature map; high-level features are upsampled or downsampled using bilinear interpolation to unify the dimensions. Feature fusion: The CA module is used to generate attention weights using the rich semantic information provided by high-level features and filter the basic information in low-level features. The specific feature fusion calculation formula is: f att =BL(T-Conv(f high )) f att =f low *CA(f att )+f att Through feature fusion, we get Among them, CA represents the channel attention module, f att is the processed high-level feature, f low is the low-level feature map, f high is the low-level feature map, f out is the fused output feature map, BL is the bilinear interpolation calculation, and T-Conv is the deconvolution calculation.

8. The infrared ship detection method based on the improved RT-DETR algorithm according to claim 1 is characterized in that: The MDST module described in T3 combines group convolution, channel shuffle, visual converter and depth-wise separable convolution techniques, and specifically includes the following features: Group convolution divides the input channels into multiple groups, and each group performs a separate convolution operation; Channel shuffling technology, by exchanging features between groups, enables the features of different groups to merge and complement each other; The visual converter splits the input features in a ratio of 3:1, and performs group convolution and channel shuffling operations on some of the features to reduce computing requirements and maintain feature diversity. It replaces the fully connected linear layer with convolution operations to reduce computing requirements and improve model performance. Depthwise separable convolution uses depthwise convolution to extract spatial features and extracts channel features through point-by-point convolution, reducing the amount of computation and improving computational efficiency. Three convolutional layers are used to complete the tasks of the fully connected linear layer, further reducing computational requirements and improving model performance.

Images