Infrared remote sensing video ship target tracking method and system based on feature fusion

Abstract

This invention discloses a method and system for tracking ship targets in infrared remote sensing video based on feature fusion, comprising the following steps: S1: Infrared remote sensing image acquisition; using a UAV equipped with a mid-wave infrared camera to capture images of the target area, obtaining a sequence of mid-wave infrared remote sensing images or a video composed of image sequences; S2: Creating a detection and tracking network; creating a detection network by improving the YOLOv10 algorithm, adding a small target detection head to the detection network, and simultaneously adding a PPA-DC module to enhance the extraction of edge features and perform feature fusion with shallow language information; creating a tracking network using DeepSORT; for the tracking part, introducing EIoU to achieve IoU matching between targets; S3: Network training and validation; training, validation, and test sets are used to train and test the network. This invention effectively improves the target detection capability in mid-wave infrared images and enhances the accuracy of ship target tracking in complex scenes.

Description

Technical Field

[0001] This invention relates to the field of infrared remote sensing information processing technology. More specifically, this invention relates to a method and system for tracking ship targets in infrared remote sensing video based on feature fusion, thereby improving the detection and tracking capabilities of ship targets in infrared remote sensing video. Background Technology

[0002] Infrared images, compared to visible light images, offer advantages such as all-weather operation and cloud / fog penetration. Compared to radar imaging, they also provide a security advantage through passive information collection, making them highly valuable in fields such as marine surveillance, maritime safety, and fisheries management. Infrared images can be further categorized by wavelength into short-wave infrared, mid-wave infrared, and long-wave infrared, with long-wave infrared being the most common. Mid-wave infrared, compared to long-wave infrared, has stronger cloud / fog penetration, making it suitable for maritime sensing in complex weather conditions. Ship targets are the primary agents in maritime activities, and automatic and accurate tracking is a core task. Therefore, this invention utilizes mid-wave infrared remote sensing video for ship target detection and tracking. Currently, detection-based multi-target tracking methods can be divided into Separate-Detection-and-Tracking (SDT) and Joint-Detection-and-Tracking (JDT) frameworks. SDT divides the target tracking task into two steps: detection and association. After the detection network detects the target, the association network uses appearance features, location, and other information to perform association. JDT integrates the two steps to a certain extent. For example, it integrates the step of extracting appearance features into the detection network, so that the detection network can directly output the classification of the target and its appearance features or directly output the intra-class classification of all targets, thereby improving the association speed.

[0003] Among target tracking methods, SDT is a more commonly used framework and has higher accuracy than JDT, but it depends on the performance of the detection network. JDT has good performance in visible light images and natural images, but target association depends on appearance features. However, ship target tracking based on mid-wave infrared remote sensing video has the following difficulties: (1) There are multi-scale targets and small target clusters, and the detection effect of general detection networks is not good; (2) There are fewer texture details in the images and the similarity between targets is high, so algorithms that rely on appearance to associate targets are difficult to associate successfully; (3) The image frame interval is long and the target displacement is large and complex, so ordinary IoU matching is difficult to perform due to the lack of overlapping areas. Summary of the Invention

[0004] To address the problems existing in the prior art, the present invention aims to provide a mid-wave infrared remote sensing video tracking algorithm based on feature fusion. First, the detection network and association network are optimized based on the two-stage algorithm of SDT type to improve the accuracy of the detection network. Second, the dependence on appearance features in the association process is reduced while ensuring smooth IoU matching, so as to improve the performance of ship target tracking in mid-wave infrared remote sensing image sequences.

[0005] To achieve the above objectives, the present invention provides a method for tracking ship targets using infrared remote sensing video based on feature fusion, the method comprising the following steps:

[0006] Step S1: Infrared remote sensing image acquisition; Using a UAV equipped with a mid-wave infrared camera, the mission area is photographed to obtain a mid-wave infrared remote sensing image sequence or a video composed of image sequences;

[0007] Step S2: Create a detection and tracking network; create a detection network by improving the YOLOv10 algorithm, add a small target detection head to the detection network, and add a PPA-DC module to enhance the detection network's extraction of edge features and feature fusion with shallow language information; create a tracking network using DeepSORT; for the tracking part, introduce EIoU to achieve IoU matching between targets;

[0008] Step S3: Network training and validation; The network is trained and tested using training, validation, and test sets.

[0009] Furthermore, in step S1, all ship targets are detected, tracked, and labeled using YOLO and MOT formats.

[0010] Furthermore, in step S2, a small target detection head is added to the YOLOv10 baseline network. The small target detection head is used to detect the high-resolution feature map with a step size of 4 as input to the detection head.

[0011] Furthermore, in addition to the small target detection head, it also includes three levels of detection heads: P3, P4, and P5, which correspond to 8x, 16x, and 32x downsampling, respectively.

[0012] Furthermore, in step S2, in order to use contour features to help the detector locate the target position, the PPA-DC module is adopted. The PPA-DC module adds differential convolution to the feature extraction process and calculates it in parallel with SPD-P. After that, feature fusion is performed, and the CA attention mechanism is used for feature enhancement.

[0013] Furthermore, differential convolution calculates the differences between pixels by introducing prior information from traditional local descriptors, and then convolves with convolution kernels to specifically capture the gradient information of pixels.

[0014] Furthermore, the PPA-DC module uses four differential convolutions and one ordinary convolution for parallel deployment.

[0015] Furthermore, differential convolution includes: (1) central differential convolution: focusing on the difference between the center and the periphery of the pixel to enhance local details; (2) angular differential convolution: capturing gradient changes at different angles and extracting features of tilted edges; (3) horizontal differential convolution: encoding the gradient in the horizontal direction and extracting edge features in the horizontal direction; (4) vertical differential convolution: encoding the gradient in the vertical direction and extracting edge features in the vertical direction.

[0016] Furthermore, in step S3, the evaluation indicators for detection are P, R, and mAP50-95; the evaluation indicators for tracking are HOTA, MOTA, MOTP, FN, and IDs.

[0017] On the other hand, the present invention provides an infrared remote sensing video ship target tracking system based on feature fusion, the system being used to implement the above-mentioned infrared remote sensing video ship target tracking method based on feature fusion.

[0018] The beneficial effects of this invention are as follows: This invention proposes an infrared remote sensing video ship target tracking method based on feature fusion, which realizes feature enhancement and fusion of edge features, effectively improving the detection capability of targets in mid-wave infrared images. At the same time, EIoU is used in the tracking association process to deal with the problems of large displacement and difficult IoU matching, effectively improving the accuracy of ship target tracking in complex scenes.

[0019] This invention proposes a YOLO-DC network based on the YOLOv10 and DeepSORT framework for infrared video tracking tasks. It optimizes the detection network by introducing a PPA-DC module to enhance edge features. Furthermore, it fuses enhanced edge features with high-resolution features at the network's neck, improving the network's ability to detect small, textured, and easily confused targets. In the tracking network, an EIoU module is added to process the bounding boxes, aiding in IoU matching and successfully addressing the problem of small targets failing to correlate successfully when the target displacement is large. Attached Figure Description

[0020] Figure 1 This is the data processing flow of the infrared remote sensing video ship target tracking method based on feature fusion according to the present invention;

[0021] Figure 2 This is a diagram of the object detection network structure;

[0022] Figure 3 This is a structural diagram of the PPA-DC module;

[0023] Figure 4This is an extended diagram;

[0024] Figure 5 This is a schematic diagram of the tracking results at a 60-degree tilt angle;

[0025] Figure 6 This is a schematic diagram of the tracking results at a 90-degree tilt angle;

[0026] Figure 7 This is a flowchart of infrared image tracking. Detailed Implementation

[0027] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0029] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0030] The following combination Figures 1-7 Specific embodiments of the present invention will be described in detail below. It should be understood that the specific embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the present invention.

[0031] This invention proposes a mid-wave infrared remote sensing video tracking algorithm based on feature fusion. First, the detection network and association network are optimized based on the two-stage algorithm of SDT class to improve the accuracy of the detection network. Second, the dependence on appearance features in the association process is reduced while ensuring smooth IoU matching, so as to improve the performance of ship target tracking in mid-wave infrared remote sensing image sequences.

[0032] The data processing flow of this invention is as follows: Figure 1 As shown, we carried out the creation of a mid-wave infrared image ship target dataset, the design of a detection and tracking network, and the training and verification of algorithms to achieve the task of tracking ship targets in mid-wave infrared remote sensing video.

[0033] The basic idea of ​​this invention is as follows: First, a mid-wave infrared image dataset is created; second, the YOLOv10 target detection network and the DeepSORT target tracking network are improved and combined; third, the improved network is used for training and validation. The invention will now be described in more detail with reference to the accompanying drawings. This invention mainly includes the following steps:

[0034] Step S1: Infrared remote sensing video image acquisition

[0035] A video composed of a sequence of mid-wave infrared remote sensing images was obtained by using a drone equipped with a mid-wave infrared camera to capture images of the mission area. All ship targets were detected, tracked, and labeled using YOLO and MOT formats. The dataset can be a video composed of infrared remote sensing image sequences or a sequence of infrared remote sensing satellite images.

[0036] Step S2: Design of Detection and Tracking Network

[0037] This invention improves the YOLOv10 algorithm and DeepSORT network structure design. Specifically, a small target detection head is added to the detection network, and a PPA-DC module is incorporated to enhance the network's edge feature extraction and feature fusion with shallow semantic information. Specifically, this invention introduces a parallelized patch-aware attention module (PPA-DC) combining differential convolution into the detection network, along with a small target detection head. The PPA-DC module has two parallel convolutional networks and one residual branch. One parallel branch is an intelligent upsampling module (SPD-P) based on local feature unrolling and fusion, extracting information such as corners, edges, lines, colors, textures, and simple shapes, which serves as shallow semantic information. The other is a differential convolution module, which uses prior information to design multiple convolutional kernels to enhance the edge features of the target. The outputs of the two parallel branches are then concatenated with the residual for dimensionality reduction, enhancing the network's ability to perceive target boundaries and improving the accuracy of object detail recognition in complex backgrounds. This approach improves the detection accuracy and recall of the detection network in mid-wave infrared images. The structure of the detection network is as follows Figure 2 As shown. For the tracking part, EIoU is introduced to achieve IoU matching between targets.

[0038] Figure 2 The relevant terms are explained below:

[0039] Conv module: The convolution module performs convolution operations on the input content to adjust the dimensions of the input information and extract features.

[0040] C2f module: The core module of the backbone network and neck network, which enhances feature representation capabilities and optimizes computational efficiency.

[0041] PPA-DC: The module proposed in this paper.

[0042] SCdown module: A downsampling module that optimizes computational cost and parameter count compared to general downsampling modules.

[0043] C2fCIB Module: Feature Enhancement Module. The optimized C2f module enhances features with better computational efficiency.

[0044] SPPF: Spatial Pyramid Pooling, which fuses multi-scale contextual information through pooling of different sizes.

[0045] PSA: Partial Self-Attention Module, an attention mechanism design that enhances the model's feature extraction capabilities.

[0046] Upsample: Upsampling enlarges the size of a low-resolution feature map into a high-resolution feature map, which facilitates multi-scale feature fusion.

[0047] Concat: This function concatenates multiple input feature maps, fusing information from different layers and scales.

[0048] S2.1YOLOv10 Small Target Detection Head

[0049] In this invention, a small target refers to a target that occupies a very small pixel area in an image, specifically, a target whose width and height are less than 5% of the total width and height of the input image. Although the YOLO network performs well in general object detection tasks, its performance is often constrained by the following challenges when dealing with small targets: (1) Feature dilution problem: During the continuous downsampling process of the backbone network, the fine-grained feature information of small targets is easily lost; (2) Receptive field mismatch: Existing detection heads (e.g., detection heads for medium or large targets) are usually based on deep feature maps, and their large receptive fields are too coarse for small targets, making it difficult to capture their precise location information. To effectively solve the above problems, this invention innovatively integrates a dedicated small target detection head into the YOLO network. This is not a simple network topology extension, but a key technical improvement aimed at accurately matching the characteristics of small targets and optimizing feature utilization efficiency. The YOLOv10 benchmark network has three levels of detection heads: P3, P4, and P5, corresponding to 8x, 16x, and 32x downsampling, respectively, to address the challenges of multi-scale problems. Statistical analysis of actual datasets revealed that the size of most targets is only 1-2% of the image width, and the features of the targets are also severely lost during the sampling process. Therefore, this invention adds a detection head based on higher resolution features, namely the P2 detection head, to YOLOv10. This detection head takes a high-resolution feature map with a step size of 4 as input.

[0050] Add a small target detection head:

[0051] 1. Selecting an appropriate feature layer: Design a new detection branch by inputting a shallow, higher resolution feature map with relatively less semantic information but more detail. Compared to feature maps used for detecting medium to large targets, this feature map has a larger spatial dimension and contains richer and finer low-level visual information, thus providing more sufficient localization details for small targets.

[0052] 2. How it works after integration: Each detection head is designed to connect to a feature map of a specific scale in the feature pyramid and is configured to detect targets of a certain size. The neck network of the network extracts feature maps of different resolutions from the processing of the backbone network. Among them, the feature maps with the highest resolution (shallow layer) are specially connected to the small target detection head because they retain rich local details and fine spatial information. This detection head is configured with a minimum anchor box to accurately capture and locate tiny targets in the image.

[0053] After inference, multiple detection heads will use non-maximum suppression post-processing to filter the results and finally output the most accurate and reliable results.

[0054] S2.2 Design of PPA-DC Edge Feature Fusion Module

[0055] Mid-wave infrared images are characterized by weak texture and strong contours. To utilize contour features to help the detector locate the target, this invention employs a PPA-DC module. The structure of the PPA-DC module is as follows: Figure 3 As shown, the PPA-DC module incorporates Differential Convolution (DEConv) into the feature extraction process and performs parallel computation with SPD-P. It then performs feature fusion and uses the CA attention mechanism for feature enhancement.

[0056] in, Figure 3 The terms shown are explained below:

[0057] PPA-DC: Parallelized Patch-Aware Attention Module Combining Differential Convolution.

[0058] SPD-P: An intelligent upsampling module based on local feature unrolling and fusion. It unrolls local region feature maps into high-resolution feature maps, then stitches these local region feature maps together, fuses this information through 1x1 convolution, and finally uses bilinear interpolation to perform resolution-enhancing feature upsampling.

[0059] conv: Convolution module; the other content within the module is the parameters of the convolution kernel.

[0060] split: Divide, the function is to separate a multidimensional feature map according to given parameters, the map is divided into two identical parts.

[0061] concat: concatenates multiple feature maps together.

[0062] DEconv: A module for differential convolution techniques, which runs multiple differential convolutions in parallel.

[0063] CA: CA attention mechanism.

[0064] Specifically, this invention introduces a parallelized patch-aware attention module (PPA-DC) combined with differential convolution into the detection network. The PPA-DC module can be described as two parallel convolutional networks and one residual branch. The first is an intelligent upsampling module (SPD-P) based on local feature unrolling and fusion, extracting information such as corners, edges, lines, colors, textures, and simple shapes. This extracted information serves as shallow semantic information. The second is a differential convolution module, which designs multiple convolutional kernels based on prior information to enhance the edge features of the target. Subsequently, the outputs of the two parallel branches are concatenated with the residual for dimensionality reduction, enhancing the network's ability to perceive target boundaries and improving the accuracy of object detail recognition in complex backgrounds.

[0065] Parallel branch one, SPD-P, takes as input a feature map from an intermediate layer of the detection network. It employs core operations including unfold + concat + 1x1 conv + Bilinear interpolation, and outputs high-resolution shallow semantic features after local information encoding and upsampling.

[0066] Parallel Branch Two: Differential Convolutional Module. Its input is the same intermediate layer feature map of the detection network as the SPD-P module. Internally, it contains a set of differential convolutional kernels designed based on prior information. The output is the enhanced target edge features.

[0067] The residual branch takes the same intermediate feature map from the detection network as the two parallel branches as its input, uses an identity mapping, and outputs a direct mapping of the original feature map.

[0068] The Fusion & Dimension Reduction Layer takes as input the outputs of the SPD-P module, the differential convolutional module, and the residual branch. It employs a concatenation operation, followed by a 1x1 convolutional layer and a CA attention mechanism to enhance features containing important information. This part of the operation combines and compresses information from various parallel branches, extracting the fused feature representation most beneficial for the object detection task; the output is a fused and dimensionality-reduced enhanced feature map.

[0069] Differential convolution refers to calculating the differences between pixels by incorporating prior information from traditional local descriptors, and then convolving it with a convolution kernel to specifically capture the gradient information (edges, contours) of pixels. It is commonly used in edge detection tasks and is therefore considered to improve the ability to extract edge features. Specifically, this invention uses four differential convolutions and one ordinary convolution in parallel deployment. The differential convolutions specifically include:

[0070] (1) Central Difference Convolution (CDC): Focuses on the difference between the center and the surrounding area of ​​a pixel, enhancing local details;

[0071] (2) Angle-differential convolution (ADC): captures gradient changes at different angles and extracts features from sloping edges;

[0072] (3) Horizontal differential convolution (HDC): Encodes the horizontal gradient and extracts the horizontal edge features;

[0073] (4) Vertical differential convolution (VDC): Encodes the vertical gradient and extracts the edge features in the vertical direction.

[0074] S2.3EIoU matching

[0075] IoU: This is an indicator that measures the degree of overlap between two regions, defined as the ratio of the area of ​​their intersection to the area of ​​their union.

[0076] EIoU: This is the extended EIoU used in this paper. It first expands the size of the detection box and then performs IoU calculation.

[0077] Because mid-wave infrared images have long frame intervals, the displacement of objects between adjacent frames is relatively large and more irregular. When the object displacement is large, the IoU value of the same object in consecutive frames is low or even 0, leading to target association failure. Expanding the detection box increases the IoU between detection boxes, as shown below:

[0078] (1)

[0079] (2)

[0080] and These are the expanded target bounding box and the predicted bounding box's width and height. The coefficient of thermal expansion is 1 / 3. and It represents the width and height of the original target bounding box and the predicted bounding box.

[0081] The expansion coefficient E is used to introduce the calculation method of EIoU. This invention designs a dynamic value for the expansion coefficient E.

[0082] E is a hyperparameter, which is determined during the algorithm's execution as follows:

[0083] (3)

[0084] in, It is an expansion cycle. Indicates the first The coefficient of thermal expansion of the wheel, It is the initial expansion coefficient. It is a step-by-step process, in which the detection box can be iteratively expanded to improve the model's adaptability to diverse targets.

[0085] and These are the model's hyperparameters, and the method for determining hyperparameters is as follows:

[0086] Set a search range The range is [0.5, 1]. The range is [0.1, 0.5], and the evaluation metric is the tracking success rate. The evaluation method is to perform a tracking task within a predetermined sequence of infrared ship images. The final determined hyperparameters are... It is 0.5. The value is 0.3.

[0087] Figure 7 A flowchart illustrating the infrared image tracking process of this invention is shown. It includes the following stages:

[0088] After the detection network detects the target, the image and the detection results are input into the tracking network. The detection network includes information such as the target's location (center point, width and height) and type.

[0089] After B enters the tracking network, it acquires appearance features. Following the acquisition of feature data, a feature-to-IoU fusion matching process is performed. The matching method is as follows:

[0090] 1. The detection results extract features and target bounding box location information;

[0091] 2. Predicted bounding box location information based on the weighted sum of the appearance features of targets in the existing trajectory.

[0092] This matching method can quickly and accurately handle the association between stationary targets and some simple moving targets. The matching results include three types: the first is an unmatched trajectory (no target to match the trajectory may be because the trajectory has ended or the target corresponding to the trajectory has been briefly eliminated); the second is an unmatched target (no suitable trajectory to match this target may be the start of a new trajectory); and the third is a matched trajectory (successfully matched).

[0093] The relevant terms are explained below:

[0094] Appearance features refer to the features extracted by the tracking network, which are stored in a data matrix (similar to feature maps in a detection network).

[0095] The matching of location information is what IoU matching refers to.

[0096] A trajectory refers to a set of targets that are associated with the same target in a time series. It includes which frame the target belongs to. Initially, there is no trajectory for the corresponding position. As the matching progresses, unmatched targets are considered as a new trajectory.

[0097] The weighted sum of appearance features refers to the appearance feature that contains all the target information on the trajectory after weighting the appearance features of the targets in the same trajectory. Weighting is a mathematical calculation.

[0098] C. If the trajectory match is successful, update the trajectory information and add it to the predicted trajectory set. Unmatched trajectories and unmatched targets will proceed to subsequent steps for more detailed EIoU matching.

[0099] D divides unmatched targets into high-confidence targets and low-confidence targets, and performs iterative EioU matching (EIoU_iterative_match) and ordinary EioU matching (EIoU_single_match) with the trajectory respectively.

[0100] EIoU matching includes two forms: EIoU_iterative_match and EIoU_single_match, each targeting detection boxes with different confidence levels. Specifically, EIoU_iterative_match is used for high-confidence detection boxes, considering both iterative EIoU and appearance feature distance when calculating the matching cost. After normalization, the smaller of the iterative EIoU and appearance feature distance is taken as the final association cost, thus achieving a balance between appearance and location. Using multi-round iterative EIoU allows for dynamic adjustment of the matching region during multiple rounds of scale expansion, reducing sensitivity to hyperparameters and maximizing the matching success rate for high-confidence targets.

[0101] Low-confidence targets are associated using EIoU_single_match, meaning only a single round of EIoU cost is used. This hierarchical strategy prioritizes matching high-confidence, highly overlapping targets, while low-confidence targets are associated gradually in subsequent stages, effectively avoiding interference from low-quality bounding boxes on the overall association results. Through this hierarchical matching process from high to low confidence, the algorithm significantly reduces matching errors caused by background noise and occlusion false detections while maintaining accurate association, achieving more robust multi-target association performance. The criteria for distinguishing between high and low confidence are hyperparameters, and the determination method is similar to... and Similarly, the range for high confidence is [0.6, 0.9], and the range for low confidence is [0.2, 0.5]. The final determined values ​​are a high confidence threshold of 0.6 and a low confidence threshold of 0.3.

[0102] The core idea of ​​EIoU is: when matching trajectories with detection boxes, the historical trajectory boxes are first expanded proportionally to form a larger-scale candidate region, thereby obtaining greater overlap. For example... Figure 4 As shown, the bounding boxes represent detection boxes in different frames. Before expansion, the IoU is 0, and it will be judged as not being the same object by IoU matching. After expansion, the IoU is not 0, and IoU matching can be performed normally.

[0103] Data processing flow: Mid-wave infrared image sequences are first input into the detection network for target detection. The PPA-DC module enhances shallow semantic information and extracts edge features, helping the model better extract this more identifiable information in mid-wave infrared images. After feature extraction, each detection head selects the feature map of the corresponding layer to detect the target location. Small target detection heads use higher resolution feature maps containing more details for detection, which helps in the extraction of small targets in the image sequence. After the detection network detects the target, the detection results are input into the tracking network. Mid-wave infrared image sequences have a large inter-frame interval, which can cause the traditional IoU detection effect to decrease or fail to work properly. Therefore, EIoU is used to expand the bounding box during the tracking matching process, which ensures successful matching and improves matching accuracy and recall.

[0104] Step S3: Network Training and Validation

[0105] This invention divides the network into training, validation, and test sets in a 7:2:1 ratio for training and testing. Evaluation metrics for detection include P, R, and mAP (50-95). Evaluation metrics for tracking include HOTA, MOTA, MOTP, FN, and IDs.

[0106] The experiment used PyTorch 2.0.1 and CUDA 11.8 as the software configuration, and an NVIDIA GeForce RTX 4060 Laptop GPU with 8GB of VRAM and an Intel Core i7-13650HX CPU as the hardware configuration. The experimental parameters were designed as follows: the detector training batch size was set to 4, with 180 training epochs. Mixed precision training was enabled during optimization, and one warmup epoch was set. No pre-trained weights were used during training. Regarding the tracker parameters, targets with a detection confidence of 0.3 or higher were tracked. The high confidence threshold was set to 0.6, the low confidence threshold to 0.3, the IoU threshold between the bounding box and the detection box to 0.2, the number of iterations for EIoU was set to 3, the initial iteration inflation rate was 0.5, the step size was 0.3, and during hierarchical matching, the low confidence EIoU iteration inflation rate was 0.9, with only one inflation epoch.

[0107] On the other hand, the infrared remote sensing video ship target tracking system based on feature fusion according to the present invention includes an image acquisition module, a detection module, and a tracking module; wherein the image acquisition module is an infrared camera, the detection module detects the target through a detection network based on an improved YOLOv10 algorithm, and the tracking module tracks the target through an improved DeepSORT tracking network.

[0108] To verify the impact of the proposed modules on detection, the models with added small target detection heads, the models with added PPA-DC modules, the complete model, and the baseline were compared. The results are shown in the table below. Compared with the baseline, the models with added small target detection heads improved mAP50 and mAP50-95 by 2.7% and 3.4%, respectively, and Recall by 2.8%, proving that small target detection heads can effectively improve the detection accuracy of small targets. The models with added PPA-DC modules improved mAP50 and mAP50-95 by 2.2% and 1.9%, respectively, and Recall by 2.1%, respectively, compared with the baseline, proving that the edge enhancement capability of the C2f-DC module can improve the detection accuracy even when the target lacks texture features. Finally, the performance of the improved YOLO algorithm in this invention improved mAP50 by 3.6%, mAP50-95 by 4.6%, and Recall by 4.7% compared with the baseline.

[0109] Table 1 Results of network ablation experiments

[0110]

[0111] The method of this invention was compared with DeepSORT, ByteTrack, and Deep-OC-SORT. The results are shown in the table below. It can be seen that the method of this invention is the best among these methods in terms of MOTA, IDF1, and HOTA metrics, fully demonstrating the advantages of the improved algorithm in mid-wave infrared video scenes. The method of this invention is only lower than DeepSORT in terms of IDs. Complex motion makes DeepSORT unable to perform well in complex scenes, treating many easily matched targets as background. The proposed tracking model has a higher FP value, mainly because the proposed model successfully correlated ships that are long-term images at the edge. These edge ships are characterized by highly variable features extracted by the ReID model. Since these targets usually only have part of their hull within the image, they are not tracked or labeled. However, the model successfully correlated them, increasing the FP value. The successful correlation of edge ships indirectly indicates that the method of this invention is not highly dependent on appearance features and also has better correlation performance.

[0112] Table 2. Results of comparative experiments on tracking networks

[0113]

[0114] To better demonstrate the effectiveness of the method of this invention, the tracking results are visualized. It can be seen that both DeepSORT and ByteTrack ignore the tracking of small targets and treat them as background, such as in frame 9 of the 90-degree scene and frames 6 and 15 of the 60-degree scene. In comparison with Deep-OC-SORT, it can be found that the method of this invention also performs better in tracking small targets. In the result of frame 47 at a 90-degree tilt angle, only the method of this invention successfully tracked the small target. In the 60-degree scene, target number 36 in frame 6 of Deep-OC-SORT changed to target number 105 in frame 15, resulting in an ID switch. In the method of this invention, target number 40 remains unchanged. Similarly, target number 35 in the method of this invention in the 60-degree scene is target number 91 in Deep-OC-SORT, and the successful association time is later.

[0115] This visualization also explains the previously mentioned issue of excessively high FN (Functional Number) metrics. In the tracking annotation, only targets with more than half of their area within the image were annotated. However, the visualization reveals detectable and trackable targets at the image edges. For these targets, the method of this invention associates more targets, such as target number 48 in the 60-degree scene, and maintains more stable associations, such as multiple targets on the lower edge of the 34th frame of the 60-degree scene.

[0116] This invention addresses infrared video tracking tasks by proposing a YOLO-DC network based on the YOLOv10 and DeepSORT framework. The detection network is optimized by incorporating a PPA-DC module to enhance edge features. Furthermore, enhanced edge features are fused with high-resolution features at the network's neck, improving the network's ability to detect small, textured, and easily confused targets. In the tracking network, an EIoU module is added to process the bounding boxes, facilitating IoU matching and successfully addressing the issue of small targets failing to correlate successfully when the target displacement is large. Comparative experiments demonstrate that the proposed method achieves superior performance.

[0117] Any process or method described in the flowcharts of this invention or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, which can be implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device. The computer-readable medium can be any medium containing a program for storage, communication, propagation, or transmission for use by the execution system, apparatus, or device, including read-only memory, magnetic disks, or optical disks.

[0118] In the description of this specification, references to terms such as "embodiment," "example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, those skilled in the art can combine or combine the different embodiments or examples described in this specification and the features therein without causing contradiction.

[0119] While embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and alterations to the above embodiments within the scope of the present invention.

Claims

1. A method for tracking ship targets using infrared remote sensing video based on feature fusion, characterized in that, The method includes the following steps: Step S1: Infrared remote sensing image acquisition; Using a UAV equipped with a mid-wave infrared camera, the mission area is photographed to obtain a mid-wave infrared remote sensing image sequence or a video composed of image sequences; Step S2: Create a detection and tracking network; create a detection network by improving the YOLOv10 algorithm, add a small target detection head to the detection network, and add a parallelized patch-aware attention module that combines differential convolution to enhance the detection network's extraction of edge features and feature fusion with shallow semantic information; create a tracking network using DeepSORT; for the tracking part, introduce EIoU to achieve IoU matching between targets; Step S3: Network Training and Validation; The network is trained and tested using training, validation, and test sets; In step S2, in order to use contour features to help the detector locate the target position, a parallelized patch perception attention module combining differential convolution is adopted. The parallelized patch perception attention module combining differential convolution incorporates differential convolution into the feature extraction process and performs parallel computation with the intelligent upsampling module based on local feature unrolling and fusion. Feature fusion is then performed, and feature enhancement is performed using the CA attention mechanism.

2. The infrared remote sensing video ship target tracking method based on feature fusion according to claim 1, characterized in that, In step S1, all ship targets are detected, tracked, and labeled using YOLO and MOT formats.

3. The infrared remote sensing video ship target tracking method based on feature fusion according to claim 2, characterized in that, In step S2, a small target detection head is added to the YOLOv10 baseline network. The small target detection head is used to input a high-resolution feature map with a step size of 4.

4. The infrared remote sensing video ship target tracking method based on feature fusion according to claim 3, characterized in that, In addition to the small target detection head, it also includes three levels of detection heads: P3, P4, and P5, which correspond to 8x, 16x, and 32x downsampling, respectively.

5. The infrared remote sensing video ship target tracking method based on feature fusion according to claim 3, characterized in that, Differential convolution calculates the differences between pixels by incorporating prior information from traditional local descriptors, and then convolves it with a convolution kernel to specifically capture the gradient information of the pixels.

6. The infrared remote sensing video ship target tracking method based on feature fusion according to claim 5, characterized in that, The parallelized patch-aware attention module, which combines differential convolutions, is deployed in parallel using four differential convolutions and one regular convolution.

7. The infrared remote sensing video ship target tracking method based on feature fusion according to claim 6, characterized in that, Differential convolution includes: (1) Central differential convolution: focuses on the difference between the center and the periphery of the pixel to enhance local details; (2) Angle differential convolution: captures gradient changes at different angles and extracts features of tilted edges; (3) Horizontal differential convolution: encodes the gradient in the horizontal direction and extracts edge features in the horizontal direction; (4) Vertical differential convolution: encodes the gradient in the vertical direction and extracts edge features in the vertical direction.

8. The infrared remote sensing video ship target tracking method based on feature fusion according to claim 7, characterized in that, In step S3, the evaluation indicators for detection are P, R, and mAP50-95; the evaluation indicators for tracking are HOTA, MOTA, MOTP, FN, and IDs.

9. An infrared remote sensing video ship target tracking system based on feature fusion, characterized in that, The system is used to implement the infrared remote sensing video ship target tracking method based on feature fusion according to any one of claims 1-8. The system includes an image acquisition module, a detection module, and a tracking module. The image acquisition module is an infrared camera, the detection module detects the target through a detection network based on an improved YOLOv10 algorithm, and the tracking module tracks the target through an improved DeepSORT tracking network.

Images