An unmanned aerial vehicle infrared and visible light ship image adaptive registration fusion method

Abstract

This invention discloses an adaptive registration and fusion method for UAV-borne infrared and visible light ship images. First, it extracts layered infrared and visible light features using a layered Transformer, then alternates between self-attention and cross-attention to reduce cross-modal feature differences. Reliable matching points are then selected using double softmax and nearest neighbor filtering. Next, an adaptive maritime strategy is constructed based on sea-line and ship thermal features to filter out unstable matching points on the sea surface. Thin-plate splines are used to complete local non-rigid registration of the image. Finally, the visible light is converted to YCbCr space, and only the luminance component is adaptively weighted and fused regionally, while the chrominance component remains unchanged. This invention solves the problem of cross-modal registration deviation caused by low texture and UAV jitter at sea, highlights ship thermal targets, produces natural image colors, and is suitable for real-time monitoring of ships at the edge of UAVs.

Description

Technical Field

[0001] This invention pertains to the field of UAV-based marine optoelectronic image processing, specifically relating to an adaptive registration and fusion method for UAV-borne infrared and visible light ship images. Background Technology

[0002] Drones equipped with visible light cameras and infrared thermal imagers are widely used for ship monitoring in scenarios such as maritime anti-smuggling, port security, and near-shore patrols. Visible light images contain complete texture and color information, making them suitable for visual interpretation; infrared images rely on thermal radiation imaging, and can reliably detect ship thermal targets at night, in foggy or backlit conditions. Registering and fusing these two types of images can simultaneously preserve detailed textures and thermal target features, improving ship identification and situational awareness capabilities.

[0003] Existing registration and fusion technologies have core defects: the marine scene has low texture and strong dynamic interference, coupled with the flight jitter of UAVs and the field of view deviation of dual sensor installation, making it difficult for traditional methods to achieve high-precision and stable registration of infrared and visible light images, resulting in poor ship target recognition after fusion.

[0004] 1. Traditional manual feature matching (SIFT, ORB, etc.) has low cross-modal feature overlap. Large areas of textureless sea surface and sky are prone to generating a large number of mismatch points. The registration model is pulled by wave disturbance, resulting in large global alignment error.

[0005] 2. Conventional global affine and homography transformations can only correct overall translation and rotation, and cannot compensate for local nonlinear deformation of images caused by UAV jitter, resulting in obvious misalignment in key areas such as ships and sea horizons;

[0006] 3. General pixel-level fusion does not distinguish between sea surface and ship semantic regions. The sea surface introduces infrared noise, and the thermal features of ships are weakened by visible light, resulting in poor visual effects of the fused image, which is not conducive to subsequent ship detection.

[0007] 4. Existing deep learning matching schemes do not incorporate maritime-specific priors, lack constraints from the sea-line and ship thermal features, have insufficient robustness in matching point selection, and have a large computational load, making them difficult to deploy on small edge computing devices such as drones.

[0008] In summary, existing technologies are not suitable for dynamic maritime vessel monitoring scenarios using UAVs. Therefore, it is necessary to design an infrared and visible light adaptive registration fusion scheme that balances registration accuracy, robustness to maritime scenarios, and real-time airborne capabilities. Summary of the Invention

[0009] To address the problems of low cross-modal registration accuracy, poor fusion effect, and inability to operate in real-time on airborne environments caused by low texture at sea and dynamic deformation of UAVs, this paper proposes an adaptive registration and fusion method for UAV-borne infrared and visible light ship images. Based on four innovations, this method achieves high-precision alignment and target enhancement fusion of infrared and visible light images in complex maritime environments, and is adapted for real-time processing at the edge of UAVs.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] An adaptive registration and fusion method for UAV-borne infrared and visible light ship images includes:

[0012] Step S1: Acquire visible light and infrared images of the sea simultaneously captured by the UAV, preprocess the two types of images, and output the corrected dual-modal image;

[0013] Step S2: A hierarchical Transformer encoder is used to extract dual-modal hierarchical features. For each layer, self-attention is first run to enhance the single-modal ship and sea antenna structure, and then cross-attention is run to establish cross-modal association between infrared and visible light features, and output a multi-layer feature map that fuses cross-modal information.

[0014] Step S3: Calculate the matching similarity based on the feature map. Perform softmax normalization on the visible light and infrared feature dimensions respectively and multiply them to obtain the bidirectional matching confidence. Combine the mutual nearest neighbor rule to filter high confidence matching points and output the preliminary matching point set.

[0015] Step S4: Based on the prior optimization of the matching point set for the maritime scene, abnormal points that deviate from the contour are eliminated by the geometric constraints of the sea-line, the weight of the infrared ship hot zone matching points is increased, the weight of the dynamic sea surface matching points is reduced, and high-precision optimized matching points are output.

[0016] Step S5: Construct a thin plate spline TPS smooth transformation model using the optimized matching points as control points. Resample the infrared image through bilinear interpolation to compensate for local nonlinear deformation and output a registered infrared image aligned with visible light.

[0017] Step S6: Convert the visible light image to the YCbCr space, separate the luminance Y and chrominance CbCr components, adaptively weight and fuse the infrared luminance information of the Y component according to the ship, sea surface and boundary area, retain the chrominance component completely, and reconstruct the color space to output the fused image.

[0018] Step S7 involves deploying the algorithms implemented in steps S1-S6 on the Jetson Xavier NX airborne platform, using GPU parallel acceleration for feature extraction and interpolation operations to achieve real-time image processing on the UAV.

[0019] Preferably, in step S1, the preprocessing includes distortion correction, grayscale equalization, and temporal difference compensation for the two types of images.

[0020] Preferably, in step S2, cross-attention uses visible light features as the Query and infrared features as the Key and Value vectors to calculate cross-modal association weights, thereby reducing the feature representation differences caused by the infrared and visible light imaging mechanisms.

[0021] Preferably, in step S3, the bidirectional matching confidence is the result of multiplying the softmax probability of the visible light dimension and the softmax probability of the infrared dimension pixel by pixel, and only matching points with a confidence greater than a preset threshold are retained.

[0022] Preferably, in step S4, the sea-line geometric constraint is obtained by extracting the sea-line straight line through edge detection and deleting invalid matching points whose vertical distance from the sea-line exceeds the threshold.

[0023] Preferably, in step S4, the weight of the matching points in the wave-reflective sea surface area is reduced to weaken the interference of sea surface dynamic disturbance on the registration model.

[0024] Preferably, in step S5, the thin plate spline transformation generates a globally smooth pixel coordinate mapping, which can correct the local image stretching and offset caused by UAV jitter and dual sensor line-of-sight deviation.

[0025] Preferably, in step S6, the infrared brightness fusion weight is increased in the ship area, the infrared weight is decreased in the sea surface area, and the image boundary is smoothed using Gaussian smoothing to achieve a gradual transition of fusion weight without splicing breaks.

[0026] Preferably, in step S7, the GPU-accelerated object includes all computational steps such as hierarchical Transformer feature extraction, thin-plate spline interpolation, and color space conversion.

[0027] An adaptive registration and fusion system for UAV-borne infrared and visible light ship images, used to perform the above-mentioned registration and fusion method, includes:

[0028] The image acquisition and preprocessing module is used to simultaneously acquire visible light and infrared images and complete preprocessing.

[0029] The hierarchical cross-modal feature matching module is used for hierarchical Transformer feature extraction, self-attention and cross-attention operations, and bidirectional high-confidence matching point filtering.

[0030] The marine matching point optimization module is used to output optimized matching control points by relying on the horizon line, ship thermal characteristics, and sea surface suppression to filter out abnormal matching points.

[0031] The TPS non-rigid registration module is used to construct a thin plate spline transformation model and use bilinear interpolation to complete local nonlinear deformation correction and pixel alignment of infrared images.

[0032] The semantic partitioning fusion module is used for YCbCr color space conversion, regional adaptive brightness weighted fusion, and outputting a fused image while preserving visible light chromaticity.

[0033] The airborne edge acceleration module is used to carry Jetson Xavier NX hardware and GPU parallel acceleration of the entire algorithm to achieve real-time airborne image processing.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. Layered cross-modal attention matching reduces the difference between infrared and visible light modes from the feature generation stage. Dual softmax bidirectional matching screening significantly reduces the cross-modal mismatch rate and solves the problem of unreliable matching in low-texture marine scenes.

[0036] 2. By integrating three prior maritime constraints—surface line, ship thermal characteristics, and sea surface suppression—the model actively filters out unstable matching points caused by waves and reflections, ensuring that the registration model is unaffected by dynamic sea surface interference and significantly improving the robustness of maritime scenarios.

[0037] 3. The non-rigid transformation of thin plate splines can compensate for local nonlinear deformations that cannot be handled by the global affine model, and significantly improve the alignment accuracy of key areas of ships and seaplanes under UAV jitter and sensor installation errors.

[0038] 4. The YCbCr color space partitioning and fusion strategy adjusts only the luminance component and retains the natural color of visible light, significantly enhancing the thermal target of ships, eliminating excess infrared noise on the sea surface, and producing natural image colors, making it suitable for manual interpretation and intelligent ship detection.

[0039] 5. The algorithm can be deployed on a small embedded platform for UAVs. After GPU acceleration, it meets the real-time processing requirements of airborne applications and is suitable for mobile monitoring scenarios of UAVs at sea. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments or prior art, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating an adaptive registration and fusion method for UAV-borne infrared and visible light ship images provided in an embodiment of the present invention.

[0042] Figure 2 This is the overall architecture diagram of an UAV-borne infrared and visible light image registration and fusion system;

[0043] Figure 3 This is a network framework diagram of an image registration and fusion system;

[0044] Figure 4 This is a schematic diagram of a layered Transformer encoder structure;

[0045] Figure 5 This is a diagram illustrating the working principle of the cross-attention mechanism;

[0046] Figure 6 It is double Schematic diagram of confidence matrix calculation;

[0047] Figure 7 This is a flowchart of adaptive matching optimization for maritime scenarios;

[0048] Figure 8 This is a schematic diagram of the sea-line detection and area division;

[0049] Figure 9 This is a schematic diagram of the thin spline interpolation registration principle;

[0050] Figure 10 yes Spatial adaptive fusion principle diagram;

[0051] Figure 11 This is a schematic diagram of adaptive weight distribution;

[0052] Figure 12 This is a flowchart of the multi-frame fusion strategy;

[0053] Figure 13 This is a comparison chart of feature matching results from different methods in infrared and visible light images of ships at sea.

[0054] Figure 14 This is a qualitative comparison of the fusion effects of different methods in infrared and visible light images of ships at sea. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0056] This invention provides an adaptive registration and fusion method for UAV-borne infrared and visible light ship images. First, image pairs are simultaneously acquired by a UAV-borne visible light camera and an infrared thermal imager, and time synchronization, distortion correction, size normalization, and radiometric normalization are performed. Then, a hierarchical cross-attention Transformer encoder is used to extract visible light and infrared features, simultaneously performing in-modal self-attention and cross-modal cross-attention at each layer to achieve synchronous optimization of feature extraction and feature matching. Finally, a dual-mode infrared image fusion method is used. Candidate matching points are obtained through confidence matrix, nearest neighbor screening, and threshold filtering. Based on these candidate matching points, this invention further introduces an adaptive matching optimization strategy for maritime scenes: establishing global geometric constraints through sea-line detection, enhancing matching points in ship areas through infrared high-temperature regions or ship detection results, and suppressing unstable matching points in sea-surface undulation regions through optical flow estimation and semantic masking. Subsequently, a thin spline transformation is solved based on the optimized matching point set to map the infrared image to the visible light image coordinate system, and this is further optimized through inverse mapping, bilinear sampling, and... Masking and boundary smoothing are used to obtain the registered infrared image. Finally, in In the color space, the fusion weights are adaptively calculated based on the ship area, the sea surface area, the transition area near the horizon, and the confidence map to generate a fused image suitable for target detection and manual interpretation.

[0057] refer to Figure 1 An adaptive registration and fusion method for UAV-borne infrared and visible light ship images specifically includes the following steps:

[0058] Step S1: Acquire visible light and infrared images of the sea simultaneously captured by the UAV, preprocess the two types of images, and output the corrected dual-modal image;

[0059] Step S2: A hierarchical Transformer encoder is used to extract dual-modal hierarchical features. For each layer, self-attention is first run to enhance the single-modal ship and sea antenna structure, and then cross-attention is run to establish cross-modal association between infrared and visible light features, and output a multi-layer feature map that fuses cross-modal information.

[0060] Step S3: Calculate the matching similarity based on the feature map. Perform softmax normalization on the visible light and infrared feature dimensions respectively and multiply them to obtain the bidirectional matching confidence. Combine the mutual nearest neighbor rule to filter high confidence matching points and output the preliminary matching point set.

[0061] Step S4: Based on the prior optimization of the matching point set for the maritime scene, abnormal points that deviate from the contour are eliminated by the geometric constraints of the sea-line, the weight of the infrared ship hot zone matching points is increased, the weight of the dynamic sea surface matching points is reduced, and high-precision optimized matching points are output.

[0062] Step S5: Construct a thin plate spline TPS smooth transformation model using the optimized matching points as control points. Resample the infrared image through bilinear interpolation to compensate for local nonlinear deformation and output a registered infrared image aligned with visible light.

[0063] Step S6: Convert the visible light image to the YCbCr space, separate the luminance Y and chrominance CbCr components, adaptively weight and fuse the infrared luminance information of the Y component according to the ship, sea surface and boundary area, retain the chrominance component completely, and reconstruct the color space to output the fused image.

[0064] Step S7 involves deploying the algorithms implemented in steps S1-S6 on the Jetson Xavier NX airborne platform, using GPU parallel acceleration for feature extraction and interpolation operations to achieve real-time image processing on the UAV.

[0065] In step S1, the preprocessing includes distortion correction, grayscale equalization, and temporal difference compensation for the two types of images.

[0066] In this embodiment, step S1 involves image acquisition and preprocessing.

[0067] refer to Figure 2 and Figure 3 The UAV-borne platform is equipped with a visible light camera and an infrared thermal imager. Synchronous data acquisition is achieved through hardware trigger lines, a unified clock, timestamp synchronization, or software alignment strategies. Before data acquisition, the intrinsic parameters of the visible light camera, the intrinsic parameters of the infrared camera, the extrinsic parameters of both cameras, and the distortion parameters must be calibrated. If there is a fixed deviation in the field of view between the two cameras, coarse registration can be performed based on the calibration parameters, followed by fine registration using subsequent learning matching and TPS (Total Photon Score).

[0068] Preprocessing includes: converting the visible light image to grayscale and scaling both images to a uniform input scale. Image boundaries are padded to meet the network downsampling requirements. The preprocessed image is denoted as... and The width of both and height Consistent.

[0069] Table 1 System composition and interrelationships

[0070] Components Structure or function relationship UAV-borne infrared and visible light image registration and fusion system It consists of a hardware acquisition platform, an edge computing unit, and a software algorithm module, and is used to complete the synchronous acquisition, registration, and fusion of infrared and visible light images. Unmanned aerial vehicle (UAV) flight platform Equipped with a multimodal imaging unit and an edge computing unit, this experiment uses a quadcopter UAV to conduct patrol monitoring of the sea area according to a preset route or mission command. Multimodal imaging unit It includes a visible light camera, an infrared thermal imager, a synchronization trigger module, calibration board parameters, and installation external parameters, used to obtain time-synchronized visible light and infrared images. Edge computing unit Jetson Xavier NX is used, with an embedded GPU for performing preprocessing, feature matching, TPS registration, image fusion, and object detection. Image preprocessing module It achieves timestamp alignment, distortion correction, grayscale conversion, size normalization, infrared dynamic range normalization, and initial coarse alignment. Hierarchical cross-attention matching module The hierarchical Transformer structure is used to output multi-scale features, similarity matrix, confidence matrix and candidate matching points. Marine Scene Adaptive Optimization Module Candidate matching points are weighted, filtered, and compensated based on the sea surface line, ship thermal area, sea surface mask, and optical flow information. TPS registration module Based on the optimized control points, the thin spline space mapping is solved, the infrared image is backsampled, and a registration map and effective mask are generated. Adaptive fusion module A semantic weight map is generated in the YCbCr space, and the brightness is fused while the visible light chromaticity is preserved to obtain a natural and target-salient fused image. Ship detection and situational awareness module The system performs ship target detection, multi-frame tracking, track estimation, heading and speed calculation, and anomaly alerts on the fused images. Remote monitoring center It receives fused images, target bounding boxes, tracks, and alarm results, and supports real-time display, historical playback, and task logging.

[0071] In step S2, cross-attention calculates cross-modal association weights using visible light features as the Query and infrared features as the Key and Value vectors, thereby reducing the feature representation differences caused by the imaging mechanisms of infrared and visible light.

[0072] In this embodiment, step S2 involves hierarchical cross-attention feature extraction and matching.

[0073] refer to Figure 4 and Figure 5 This invention employs a hierarchical Transformer encoder to extract cross-modal features. The encoder comprises four stages, each including a same-modal self-attention module, a cross-modal cross-attention module, a feedforward neural network, and an optional downsampling module. Unlike methods that extract features independently first and then calculate similarity, this invention introduces cross-modal information during the feature extraction stage, enabling the visible light branch to perceive infrared thermal structures and the infrared branch to perceive visible light textures and hull contours, thereby obtaining feature representations more suitable for cross-modal matching. Detailed parameters are shown in Table 2.

[0074] Table 2 Encoder Four-Stage Parameters

[0075]

[0076] For the Each stage, the input features are and First, perform layer normalization and self-attention operations respectively: ; ;

[0077] Then, cross-attention is performed between the two modalities. Taking the visible light branch as an example, the visible light features are used as the query, and the infrared features are used as the key and value, resulting in the cross-modal enhanced visible light features; the infrared branch is calculated in reverse order.

[0078] ;

[0079] ;

[0080] Finally, feature refinement is achieved through a feedforward network and residual connections. If the current stage is not the last layer, spatial resolution is reduced and the number of channels is increased through convolutional downsampling or patch merging, thereby forming a multi-scale representation.

[0081] In step S3, the confidence level of bidirectional matching is the result of multiplying the softmax probability of the visible light dimension and the softmax probability of the infrared dimension pixel by pixel, and only matching points with a confidence level greater than a preset threshold are retained.

[0082] In this embodiment, step S3 involves cross-modal feature matching.

[0083] After obtaining the features from the final layer or multi-scale fusion, the visible light and infrared features are flattened into a sequence of feature points and then processed. Normalization. Let the normalized visible light characteristics be... Infrared characteristics are Then the similarity matrix Defined as: ;

[0084] To obtain a matching confidence score with bidirectional consistency, refer to Figure 6 This invention employs double softmax calculation:

[0085]

[0086] in This is a temperature parameter, with a value of 0.1. This indicates that infrared candidate points are normalized based on visible light feature points. This indicates that visible light candidate points are normalized using infrared feature points as a condition. After multiplying the probabilities in both directions, only candidate point pairs with high confidence in both directions are retained.

[0087] Candidate matching point selection includes mutual nearest neighbor filtering, confidence threshold filtering, and spatial homogenization sampling. Mutual nearest neighbor is used to eliminate fuzzy matches of one-to-many or many-to-one nature; the confidence threshold is used to eliminate low-confidence matches; and spatial homogenization sampling is used to avoid matching points being overly concentrated in local hot spots on the ship or in the central region of the image.

[0088] In step S4, the sea-line geometric constraint is used to extract the sea-line straight line through edge detection, and invalid matching points whose vertical distance from the sea-line exceeds a threshold are deleted. In step S4, the weight of matching points in the wave-reflective sea surface area is reduced to weaken the interference of dynamic sea surface disturbances on the registration model.

[0089] In this embodiment, step S4 involves adaptive matching optimization of the maritime scenario.

[0090] refer to Figure 7 and Figure 8 In maritime scenarios, not all matching points are equally reliable. This invention uses maritime semantic information to weight and filter candidate matching points, making the registration model focus more on stable structures and ship target areas.

[0091] 1) Detection and constraint of sea surface area

[0092] Edge detection is performed on the visible light image, and the sea-line is obtained using semantic segmentation. After detecting the sea-line, the image is divided into a sky region, a sea surface region, and a transition region near the sea-line. For matching points located near the sea-line, their positional relationship relative to the sea-line must remain consistent in both modes; if a matching point is located below the sea-line in the visible light image but above it in the infrared image mapping, and the deviation exceeds a threshold, it is considered an abnormal match.

[0093] 2) Enhanced thermal characteristics of ships

[0094] In infrared images, ship engine rooms, smokestacks, personnel, and certain metal structures typically appear as areas of high temperature or significant thermal variation. This invention generates ship thermal area masks based on a ship target detection network. .right The candidate matching points within the region are weighted more, local contrast enhancement is performed in the infrared high-temperature region, and more feature points are allowed to be retained in the enhanced region.

[0095] 3) Compensation for sea surface wave motion

[0096] The sea surface region exhibits dynamic waves and reflections, easily generating short-term, non-rigid, and unstable local textures. For matching points in the sea surface region, this invention obtains the local motion direction and amplitude through optical flow estimation from adjacent frames; if the optical flow direction dispersion in the region is large or the amplitude exceeds a threshold, the weight of the matching points in that region is reduced. For sea surface points with a consistent motion trend, compensation is performed along the optical flow direction before participating in model estimation.

[0097] 4) Matching point weighting function

[0098] Let the first The initial confidence level of each candidate matching point is: The indicator variables located in the ship area, the area near the horizon, and the sea surface area are respectively , and Then the following weighted confidence level can be constructed:

[0099] ;

[0100] in , and These are the weighting coefficients. This is an index of sea surface wave instability. If... If a matching point is below a threshold or has an inconsistent geometric relationship with its surrounding matching points, it is discarded.

[0101] In step S5, the thin plate spline transformation generates a globally smooth pixel coordinate mapping, which can correct local image stretching and offset caused by UAV jitter and dual sensor line-of-sight deviation.

[0102] In this embodiment, step S5 involves thin spline interpolation and registration.

[0103] refer to Figure 9 After optimization and filtering, a set of control point pairs was obtained. and ,in It can represent the coordinates of a target point in a visible light image. It can represent the coordinates of the source point in an infrared image. The goal is to find a smooth mapping function. , making as close as possible At the same time, the overall bending energy is minimized.

[0104] ;

[0105] The present invention preferably employs a reverse mapping strategy, that is, for each pixel position in the visible light target image coordinate system... ,pass Mapping to obtain its corresponding coordinates in the infrared source image Then, bilinear interpolation is used to sample the grayscale of the infrared image. Reverse mapping can avoid the holes and mosaic effects in forward mapping.

[0106] For coordinates mapped outside the infrared image, an invalid mask is generated. The valid mask is then Gaussian blurred to obtain... A boundary transition map is used to avoid abrupt changes in the edges of the registered infrared image affecting the fusion effect.

[0107] In step S6, the infrared brightness fusion weight is increased in the ship area and decreased in the sea surface area. The image boundary is smoothed using Gaussian smoothing to achieve a gradual transition of fusion weight without stitching breaks.

[0108] In this embodiment, step S6 involves YCbCr spatial adaptive fusion.

[0109] refer to Figure 10 and Figure 11 In obtaining registered infrared images Then, the visible light image Transform from GRB space to YCbCr space to obtain , and Normalize the registered infrared image to the same standard. The same brightness is denoted as The visible light chromaticity component is preserved during fusion. and Adaptive fusion is performed only on the luminance component:

[0110]

[0111] ,

[0112] in To fuse the weighted map, it can be jointly calculated based on the ship area, sea surface area, transition area near the horizon, registration confidence, and effective infrared mask. In one embodiment, Defined as:

[0113] ;

[0114] in Based on the weights, For ship area mask, The distance from the pixel to the sea surface line. For the sea surface area mask, This is the Gaussian transition width. The fusion weights also need to be combined with... Effective mask multiplication avoids incorrect merging of invalid boundary regions.

[0115] Preferably, in step S7, the GPU-accelerated object includes all computational steps such as hierarchical Transformer feature extraction, thin-plate spline interpolation, and color space conversion.

[0116] In this embodiment, step S7 involves real-time accelerated deployment at the edge.

[0117] The algorithms implementing steps S1-S6 are deployed on the Jetson Xavier NX airborne edge computing platform. The GPU is used to accelerate Transformer feature extraction and thin-plate spline interpolation operations in parallel, reducing algorithm latency and meeting the real-time image processing requirements during UAV flight.

[0118] To reduce single-frame noise and enhance continuous monitoring stability, this invention can also introduce a multi-frame fusion strategy. First, the inter-frame difference and optical flow field between adjacent frames are calculated, dividing the image into static and dynamic regions. For static regions, time-weighted averaging or sliding window fusion is used; for dynamic regions, fusion is performed after alignment along the optical flow direction to avoid motion blur caused by ship movement. v;

[0119] in For the first The image after motion compensation. The weighting is time-based. Images closer to the current frame have higher weights, and frames with higher detection confidence have higher weights.

[0120] To train the hierarchical cross-attention matching network in this invention to establish a reliable correspondence between infrared and visible light images of ships carried by UAVs, a joint loss function is employed. This loss function directly corresponds to the aforementioned cross-modal feature matching and... Registration process.

[0121] In the aforementioned step S3, the visible light characteristics have already been used... and infrared features The similarity matrix was calculated. and through double Normalization yields the matching confidence matrix Let the set of supervised matching points be... ,in Represents the first in a visible light image The feature point and the first feature point in the infrared image These feature points constitute the supervised correspondence. The cross-modal matching loss is defined as: ;

[0122] in, Indicates the first The visible light feature point and the first Matching confidence between infrared feature points A very small constant to prevent overflow in logarithmic calculations.

[0123] This loss is used to improve the response value of the supervised matching points in the confidence matrix, enabling the network to learn the modality-invariant correspondence between infrared and visible light images. Because this invention employs a hierarchical cross-attention encoder, it enhances the single-modal structure representation through self-attention during the feature extraction stage and establishes cross-modal associations through cross-attention. Therefore, this matching loss enables the network to output more stable matching points for hull structures, structures near the sea-line, and significant hot target regions.

[0124] To make the matching points output by the network more suitable for subsequent thin spline interpolation registration, this invention introduces a registration consistency loss. Let the source control points in the infrared image be... The target control points in the visible light image are The thin spline mapping function is The registration consistency loss is defined as: ;

[0125] in, Indicates infrared control point The position of the thin spline mapped in the visible light image coordinate system. This represents the location of the corresponding visible light control point. This loss is used to constrain the infrared control points to be as close as possible to their corresponding visible light control points after spatial mapping, thereby improving the spatial consistency between the registered infrared and visible light images.

[0126] The total loss function of this invention is defined as: ;

[0127] in, Used to improve the confidence of true matching points between infrared and visible light features. Used to constrain matching points Spatial alignment error after mapping This is the balance coefficient between cross-modal matching loss and registration consistency loss, which is set to 0.5 here.

[0128] Through the aforementioned loss function design, this invention enables the hierarchical cross-attention matching network to simultaneously meet the requirements of feature matching reliability and geometric registration consistency. The network output matching points after training not only have high matching confidence but also serve as stable control points for thin spline interpolation registration, thereby improving the registration accuracy of UAV-borne infrared and visible light ship images and enhancing the alignment of ship edges, salience of thermal targets, and overall visual interpretation in subsequent fused images.

[0129] This invention also provides an UAV-borne infrared-visible light ship image adaptive registration and fusion system for performing the above-mentioned registration and fusion method, including:

[0130] The image acquisition and preprocessing module is used to simultaneously acquire visible light and infrared images and complete preprocessing.

[0131] The hierarchical cross-modal feature matching module is used for hierarchical Transformer feature extraction, self-attention and cross-attention operations, and bidirectional high-confidence matching point filtering.

[0132] The marine matching point optimization module is used to output optimized matching control points by relying on the horizon line, ship thermal characteristics, and sea surface suppression to filter out abnormal matching points.

[0133] The TPS non-rigid registration module is used to construct a thin plate spline transformation model and use bilinear interpolation to complete local nonlinear deformation correction and pixel alignment of infrared images.

[0134] The semantic partitioning fusion module is used for YCbCr color space conversion, regional adaptive brightness weighted fusion, and outputting a fused image while preserving visible light chromaticity.

[0135] The airborne edge acceleration module is used to carry Jetson Xavier NX hardware and GPU parallel acceleration of the entire algorithm to achieve real-time airborne image processing.

[0136] To verify the effectiveness of the UAV-borne multimodal image adaptive registration and fusion method described in this invention in maritime vessel monitoring scenarios, this embodiment uses the DMSD maritime infrared and visible light vessel image dataset, with 1730 pairs of infrared and visible light vessel images as the training set and 433 pairs as the test set. Training and testing are conducted using real maritime scenarios observed by a quadcopter UAV equipped with a visible light camera and an infrared thermal imager. In the training data construction phase, this invention employs a combination of synthetic geometric transformation and artificial sparse keypoints to construct a supervised matching point set. For aligned DMSD infrared and visible light image pairs, a dense supervised correspondence is generated by applying a known random geometric transformation to one of the images. Simultaneously, 2000 pairs of real maritime vessel images are manually labeled with hull stability keypoints to supplement the real cross-modal sparse supervision.

[0137] In this embodiment, the Adam optimizer is used for training, and the initial learning rate and batch size are set to [values ​​to be filled in]. 4. This invention is implemented based on the PyTorch framework, and all training and testing are performed on a single NVIDIA RTX 4090 GPU. Temperature normalization uses 5% and 95% percentile clipping; dual... Temperature parameters The threshold is set to 0.1; the matching confidence threshold is set to 0.2; approximately 200 to 500 candidate matching points are retained per frame; the sea-line detection adopts a combination of edge detection and Hough line detection. Smoothing coefficient Take 0.01; In the fusion process, the infrared weight for the ship area is set to 0.7, the infrared weight for the sea surface area is set to 0.3, and the infrared weight near the horizon is used... Gaussian transition for pixels.

[0138] To comprehensively evaluate the performance of this invention, experiments were conducted on image registration and image fusion. For registration performance, the average initial number of matching points, average number of RANSAC in-frame points, average in-frame rate, and average time consumption were used as the main evaluation metrics. Specifically, the average initial number of matching points reflects the ability of different methods to establish candidate correspondences between infrared and visible light images; the average number of RANSAC in-frame points reflects the number of valid matching points retained after geometric consistency verification; the average in-frame rate measures the reliability and resilience to false matching of the matching results; and the average time consumption evaluates the engineering applicability of the method in UAV platforms or real-time monitoring scenarios.

[0139] In terms of fusion performance, this invention selects correlation coefficients that are strongly associated with maritime vessel monitoring tasks. Structural similarity Comprehensive information maintenance indicators and contrast retention index As an evaluation metric, it is used to evaluate the information correlation between the fused image and the source image, the ability to preserve the hull structure and edge details, the ability to express complementary information in infrared and visible light, and the target contrast and visual clarity of the fused image.

[0140] Specifically, this invention includes the following types of comparative experiments. The first type is a comparison of registration methods, selecting LGHD, RIFT, and SuperPoint as comparison methods. These three methods represent traditional local gradient descriptors, multimodal frequency domain feature descriptors, and deep learning feature point methods, respectively. The experiments are conducted under the same image input, the same geometric estimation model, and the same evaluation metrics to compare the number of matches, the proportion of interior points, the registration error, and the operational efficiency of different methods in cross-modal ship scenarios.

[0141] The second category is a comparison of fusion methods, including unregistered direct fusion methods CDDFuse, DCEvo, and FreqGAN, which directly input spatially misaligned infrared and visible light images into the fusion algorithm to verify the necessity of registration for fusion quality. It also includes registration-fusion cascade methods, namely, methods that first register and then fuse, namely CrossRAFT[8]+CDDFuse, CrossRAFT+DCEvo, and CrossRAFT+FreqGAN, which are used to verify the overall advantages of the adaptive registration-fusion process in this invention.

[0142] Table 3 Comparison of registration quantification results using different methods

[0143]

[0144] As shown in Table 3, the LGHD method has a relatively low average number of initial matching points and RANSAC inliers, indicating its limited effective matching capability in infrared and visible light images. While the RIFT method can obtain more initial matching points, its average inlier rate is only 11.83%, indicating a high mismatch rate and insufficient geometric consistency. The SuperPoint method has a faster running speed, but its average number of initial matching points, inliers, and inlier rate are all lower than the method described in this invention, indicating that its matching stability in cross-modal infrared-visible light scenarios is still insufficient.

[0145] In comparison, the method of this invention achieved the highest average initial matching point count, the highest average RANSAC in-line point count, and the highest average in-line point rate on 943 pairs of infrared and visible light test images, reaching 230.76%, 97.45%, and 44.11%, respectively. This result demonstrates that the method of this invention can obtain a greater number of cross-modal matching points with stronger geometric consistency in UAV-borne maritime vessel monitoring scenarios, providing a more stable and reliable correspondence for subsequent image registration and fusion. Furthermore, the average processing time of the method of this invention is 0.154 s, significantly lower than LGHD and RIFT, demonstrating good efficiency for engineering applications.

[0146] like Figure 12 and Figure 13 As shown, a visual comparison of the feature matching performance of different methods in infrared and visible light images of ships at sea is presented. The left side of the figure is the visible light image, and the right side is the infrared image; the line connecting the two images represents the feature correspondence established between them. Due to the presence of weak sea surface texture, dark areas of the ship hull, differences in infrared thermal radiation, background interference from the sea horizon, and local occlusion of the ship structure in maritime scenes, there are significant modal differences between infrared and visible light images. This leads to traditional feature matching methods being prone to insufficient matching points, uneven distribution of matching regions, and mismatches.

[0147] The comparison results show that although SuperPoint can obtain a certain number of matching points near the hull edge, sea-line, and shoreline, the overall effective matching points are few and mostly concentrated in local strong edge areas, making it difficult to cover the main hull structure and distant background areas. The sparse distribution of LGHD and RIFT matching lines indicates that the method has limited adaptability to cross-modal grayscale differences in infrared and visible light. At the same time, LGHD matching points are concentrated near the sea-line or hull edge, making them susceptible to the effects of sea surface ripples, reflections, and changes in infrared imaging brightness, thus reducing the stability of subsequent geometric transformation estimation.

[0148] In contrast, the method of this invention can obtain richer matching points on the hull outline, deck structure, mast area, near the horizon, and distant background structures. The matching lines are more densely distributed and more consistent in direction, indicating that the obtained matching points have better geometric consistency. Especially when the infrared image texture is weak and the local brightness variation of the visible light image is significant, the method of this invention can still maintain a large number of stable corresponding points, which is beneficial to improving the registration reliability between UAV-borne infrared and visible light images.

[0149] Table 4. Quantitative comparison results of fusion performance of different methods

[0150]

[0151] Table 4 presents the quantitative evaluation results of different infrared and visible light image fusion methods on marine vessel image data. The comparison methods are divided into two categories: one is the cascaded registration fusion method, which first uses CrossRAFT for cross-modal image registration, and then connects to fusion methods such as CDDFuse, DCEvo, and FreqGAN; the other is the direct fusion method, which does not perform explicit registration and directly inputs the infrared and visible light images into CDDFuse, DCEvo, and FreqGAN for fusion. The last row of the table represents the method of this invention. As shown in Table 4, the method of this invention achieved superior overall performance in the marine vessel image fusion experiment. (The correlation coefficient is not specified in the original text.) In terms of performance metrics, the method of this invention outperforms both the cascaded quasi-fusion method and the direct fusion method. This result demonstrates that the fused image generated by this invention has a stronger information correlation with the source image, and can more fully preserve the complementary information in the infrared and visible light images. Regarding structure preservation capability, the method of this invention… This is the highest value among all comparison methods. This indicates that the method employs hierarchical cross-attention matching, adaptive matching optimization for maritime scenarios, and... After registration, the infrared and visible light images exhibit better spatial consistency in areas such as the hull outline, deck structure, mast edges, and near the horizon, thereby reducing structural misalignment, edge ghosting, and local artifacts during the fusion process. This improves overall information preservation metrics. The proposed method achieves a resolution of 0.9547, which is also the best result among all methods. For maritime vessel monitoring tasks, this result demonstrates that the fused image of the proposed method not only possesses good visual naturalness but also highlights ship thermal targets and key structural areas, which is more beneficial for subsequent target detection, manual interpretation, and situational awareness. Regarding contrast preservation index... While the direct fusion method is similar to the method of this invention, its... and The results are significantly lower than those of the present invention, indicating that although it can enhance contrast locally, it lacks effective registration constraints and is prone to problems such as hull edge misalignment and structural inconsistencies. The present invention, while maintaining high contrast, also achieves better correlation, structural similarity, and overall information preservation capabilities.

[0152] Comparing different methods, direct fusion methods, while simple in process, are prone to ghosting or artifacts near hull edges, masts, deck structures, and sea-line areas because they do not fully address the spatial misalignment between infrared and visible light images. Cascaded quasi-fusion methods improve some misalignment by registering first and then fusing, but their registration and fusion processes are relatively independent, and the fusion result is still affected by registration errors and cross-modal feature instability. In contrast, this invention establishes cross-modal feature correspondences through a hierarchical cross-attention encoder, combined with sea-line constraints, ship thermal enhancement, sea wave compensation, and... Non-rigid registration improves the spatial consistency between the infrared and visible light images before fusion; simultaneously, it utilizes... The spatially adaptive fusion strategy preserves visible light chromaticity information and enhances infrared brightness contribution, thereby achieving optimal results across multiple metrics. Therefore, the method of this invention... , , and The optimal values ​​were achieved across all four indicators, indicating that it can achieve a good balance in maintaining complementary information from the source images, maintaining the consistency of the ship's structure, enhancing target contrast, and improving the interpretability of the fused images. It is suitable for practical application scenarios such as UAV-borne maritime vessel monitoring, maritime search and rescue, near-shore inspection, and maritime security.

[0153] like Figure 14 As shown, this invention further qualitatively compares the visual effects of different methods in the fusion of infrared and visible light images of ships at sea. In the figure, VIS represents the visible light image, IR represents the infrared image, and Gradient difference represents the gradient overlay of infrared and visible light, to facilitate viewing the geometric misalignment of the image pairs. From... Figure 14(a) through (g) represent different methods. The figures show typical scenes including near-shore vessels, small boats viewed from a tilted angle, near-shore buildings, and complex sea backgrounds, covering situations such as backlighting, low-texture sea surfaces, distant targets, partial hull occlusion, and significant infrared thermal targets. The red and green boxes in the figures are used to mark the hull structure, distant background, text areas, and local detail areas. The magnified views of the details are used to observe the differences between the different methods in edge alignment, texture preservation, and artifact suppression.

[0154] Qualitative results show that direct fusion methods without effective registration are prone to structural misalignment, ghosting, and local blurring at ship edges, masts, horizon lines, and distant background areas. Particularly when sea waves are significant, the UAV's viewpoint is tilted, or there are local geometric discrepancies between infrared and visible light images, direct fusion methods often forcibly superimpose misaligned infrared thermal information and visible light textures, resulting in unclear ship outlines and shifted background edges, affecting subsequent manual interpretation and target detection.

[0155] Cascaded registration and fusion methods can improve image misalignment to some extent. However, because the registration and fusion steps are relatively independent, and general registration methods are not adaptable enough to low-texture areas at sea, reflective water surfaces, and cross-modal thermal radiation differences, some local areas still suffer from blurred edges, loss of detail, or unnatural colors. For example, in the magnified results of distant mountains, near the horizon, and ship outlines, structural discontinuities or weakened textures can still be observed.

[0156] In contrast, the method of this invention achieves more stable and natural fusion results in multiple scenarios. First, in the ship's main body area, this invention can better preserve the ship's outline, deck structure, mast edges, and local textures in the visible light image, while introducing thermally salient information from the infrared image, making the ship target more prominent. Second, in stable background areas such as the horizon, distant mountains, and shoreline buildings, the method of this invention has better edge alignment, stronger structural continuity in magnified local areas, and fewer ghosting and misalignment phenomena. Third, in the sea surface area, this invention, through adaptive matching optimization of the sea scene and semantic weight fusion, can suppress the excessive introduction of wave textures and infrared noise, making the background of the fused image smoother and more natural.

[0157] For scenarios containing text, shoreline facilities, or small targets at a distance, the method of this invention can also maintain the readability of details in visible light images, avoiding the obscuring of text, edges, or background structures due to excessive infrared brightness. Simultaneously, in the thermal target area of ​​a ship, the method of this invention can enhance the contribution of infrared information, making high-temperature areas of the ship's hull and prominent targets clearer, which is beneficial for applications such as maritime search and rescue, detection of abnormal vessels, and nighttime monitoring.

[0158] The comprehensive qualitative results show that the method of this invention achieves a balance between highlighting infrared thermal targets and preserving visible light structural details in terms of visual effect, effectively reducing edge ghosting and local artifacts caused by unregistered fusion. Compared with the comparative methods, the fused image generated by this invention has better hull edge sharpness, background structure consistency, and overall visual naturalness, making it more suitable for target identification, manual interpretation, and situational awareness tasks in UAV-borne maritime vessel monitoring.

[0159] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An adaptive registration and fusion method for UAV-borne infrared and visible light ship images, characterized in that, include: Step S1: Acquire visible light and infrared images of the sea simultaneously captured by the UAV, preprocess the two types of images, and output the corrected dual-modal image; Step S2: A hierarchical Transformer encoder is used to extract dual-modal hierarchical features. For each layer, self-attention is first run to enhance the single-modal ship and sea antenna structure, and then cross-attention is run to establish cross-modal association between infrared and visible light features, and output a multi-layer feature map that fuses cross-modal information. Step S3: Calculate the matching similarity based on the feature map. Perform softmax normalization on the visible light and infrared feature dimensions respectively and multiply them to obtain the bidirectional matching confidence. Combine the mutual nearest neighbor rule to filter high confidence matching points and output the preliminary matching point set. Step S4: Based on the prior optimization of the matching point set for the maritime scene, abnormal points that deviate from the contour are eliminated by the geometric constraints of the sea-line, the weight of the infrared ship hot zone matching points is increased, the weight of the dynamic sea surface matching points is reduced, and high-precision optimized matching points are output. Step S5: Construct a thin plate spline TPS smooth transformation model using the optimized matching points as control points. Resample the infrared image through bilinear interpolation to compensate for local nonlinear deformation and output a registered infrared image aligned with visible light. Step S6: Convert the visible light image to the YCbCr space, separate the luminance Y and chrominance CbCr components, adaptively weight and fuse the infrared luminance information of the Y component according to the ship, sea surface and boundary area, retain the chrominance component completely, and reconstruct the color space to output the fused image. Step S7 involves deploying the algorithms implemented in steps S1-S6 on the Jetson Xavier NX airborne platform, using GPU parallel acceleration for feature extraction and interpolation operations to achieve real-time image processing on the UAV.

2. The UAV-borne infrared-visible light ship image adaptive registration and fusion method according to claim 1, characterized in that, In step S1, the preprocessing includes distortion correction, grayscale equalization, and temporal difference compensation for the two types of images.

3. The UAV-borne infrared-visible light ship image adaptive registration and fusion method according to claim 1, characterized in that, In step S2, cross-attention calculates cross-modal association weights using visible light features as the Query and infrared features as the Key and Value vectors, thereby reducing the feature representation differences caused by the imaging mechanisms of infrared and visible light.

4. The UAV-borne infrared-visible light ship image adaptive registration and fusion method according to claim 1, characterized in that, In step S3, the confidence level of bidirectional matching is the result of multiplying the softmax probability of the visible light dimension and the softmax probability of the infrared dimension pixel by pixel, and only matching points with a confidence level greater than a preset threshold are retained.

5. The UAV-borne infrared-visible light ship image adaptive registration and fusion method according to claim 1, characterized in that, In step S4, the sea-line geometric constraint is used to extract the sea-line straight line through edge detection and delete invalid matching points whose vertical distance from the sea-line exceeds the threshold.

6. The UAV-borne infrared-visible light ship image adaptive registration and fusion method according to claim 1, characterized in that, In step S4, the weight of the matching points in the wave-reflective sea surface area is reduced to weaken the interference of sea surface dynamic disturbance on the registration model.

7. The UAV-borne infrared-visible light ship image adaptive registration and fusion method according to claim 1, characterized in that, In step S5, the thin plate spline transformation generates a globally smooth pixel coordinate mapping, which can correct local image stretching and offset caused by UAV jitter and dual sensor line-of-sight deviation.

8. The UAV-borne infrared-visible light ship image adaptive registration and fusion method according to claim 1, characterized in that, In step S6, the infrared brightness fusion weight is increased in the ship area and decreased in the sea surface area. The image boundary is smoothed using Gaussian smoothing to achieve a gradual transition of fusion weight without stitching breaks.

9. The UAV-borne infrared-visible light ship image adaptive registration and fusion method according to claim 1, characterized in that, In step S7, the GPU-accelerated object includes all computational steps such as hierarchical Transformer feature extraction, thin-plate spline interpolation, and color space conversion.

10. An adaptive registration and fusion system for UAV-borne infrared and visible light ship images, characterized in that, The method for implementing the UAV-borne infrared-visible light ship image adaptive registration and fusion method according to any one of claims 1 to 9 includes: The image acquisition and preprocessing module is used to simultaneously acquire visible light and infrared images and complete preprocessing. The hierarchical cross-modal feature matching module is used for hierarchical Transformer feature extraction, self-attention and cross-attention operations, and bidirectional high-confidence matching point filtering. The marine matching point optimization module is used to output optimized matching control points by relying on the horizon line, ship thermal characteristics, and sea surface suppression to filter out abnormal matching points. The TPS non-rigid registration module is used to construct a thin plate spline transformation model and use bilinear interpolation to complete local nonlinear deformation correction and pixel alignment of infrared images. The semantic partitioning fusion module is used for YCbCr color space conversion, regional adaptive brightness weighted fusion, and outputting a fused image while preserving visible light chromaticity. The airborne edge acceleration module is used to carry Jetson Xavier NX hardware and GPU parallel acceleration of the entire algorithm to achieve real-time airborne image processing.

Images