A method and system for real-time alignment of visible and infrared images taken by a drone

Abstract

The application discloses the technical field of image processing and relates to a UAV aerial visible light infrared image real-time alignment method and system. The UAV aerial visible light infrared image real-time alignment method comprises three steps of UAV aerial visible light and infrared image preprocessing, visible light and infrared image alignment deformation field training solving and constant deformation field deployment for real-time aerial dual-light image alignment. The UAV aerial visible light infrared image real-time alignment method solves the problem that the prior art is difficult to balance in efficiency and accuracy, and is suitable for UAV multi-modal image fusion, target detection and other downstream tasks.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method and system for real-time alignment of visible light and infrared images taken by unmanned aerial vehicles. Background Technology

[0002] Effective fusion of visible light and infrared images is crucial for robust perception in complex environments for unmanned aerial vehicles (UAVs), and accurate alignment of the dual-modal images is a fundamental prerequisite. However, the visible light and infrared sensors of UAVs differ in their fields of view (FOV). Visible light sensors typically have a wider FOV, while infrared sensors have a relatively narrower FOV. This results in a resolution difference between visible light images (e.g., 1920×1080) and infrared images (e.g., 640×512), making accurate visible light-infrared image alignment a significant challenge.

[0003] Current research on the alignment of visible light and infrared images captured by UAVs is limited. Mainstream bimodal alignment methods primarily focus on general scenarios, relying on local feature matching and then using parametric transformation estimation to align the two modalities. These methods simulate camera parameter effects through low-degree-of-freedom representations (such as 3×3 homography matrices), making them highly sensitive to erroneous feature matching. While some direct methods can achieve pixel-level alignment, they lack the generalization capabilities required for UAV applications.

[0004] Alignment based on feature matching is mainly divided into three categories: traditional feature matching, sparse learnable feature matching, and dense learnable feature matching. Traditional feature matching methods, such as SIFT and ORB, have limitations such as sensitivity to extreme viewpoints and lighting changes, and performance degradation in textured regions. Sparse learnable feature matching methods, such as SuperGlue, LightGlue, and OmniGlue, rely on local keypoint detection and are limited in areas with insufficient texture. Dense learnable feature matching methods, such as LoFTR and ELOFTR, can achieve pixel-level correspondence, but have high computational costs or suffer from domain discrepancies. Therefore, there is an urgent need for a real-time alignment method and system that can balance efficiency and accuracy and is suitable for UAV aerial photography of visible and infrared images. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for real-time alignment of visible light and infrared images taken by UAVs, which solves the technical problem of difficulty in balancing efficiency and accuracy in UAV visible light-infrared image alignment. This invention can realize the alignment of visible light and infrared images taken by UAVs, and has higher professionalism and deployability.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] This invention provides a method for real-time alignment of visible light and infrared images captured by unmanned aerial vehicles, comprising the following steps:

[0008] S1. Preprocessing of visible light and infrared images captured by UAV: ​​Cropping and downsampling the visible light images captured by UAV to match their size with the infrared images;

[0009] S2. Training and solving the deformation field aligned with visible light and infrared images: Input the preprocessed visible light and infrared images into the PDFNet network, and obtain the deformation field through network training. The deformation field converges to a constant state independent of the input, i.e., a constant deformation field.

[0010] S3. Constant Deformation Field Deployment for Real-Time Aerial Dual-Light Image Alignment: A constant deformation field is deployed on the UAV to preprocess the real-time captured visible light image, and then the constant deformation field is applied to achieve alignment with the infrared image.

[0011] Furthermore, S1 includes the following steps:

[0012] S11. Preprocess the visible light image collected by the UAV: ​​crop the pixels of the visible light image from 1920×1080 to 1920×1024 and remove unnecessary edge areas;

[0013] S12. Then, the pixels of the cropped visible light image are downsampled to 640×512 to make its size consistent with the infrared image taken by the drone.

[0014] Furthermore, S2 includes the following steps:

[0015] S21. The preprocessed visible light image and infrared image are input into the PDFNet network. The PDFNet network processes the two inputs simultaneously through a feature encoder, calculates the local feature similarity to obtain a correlation map, and then the decoder predicts the deformation field. The displacement of each pixel in the deformation field is used to rearrange the pixels of the visible light image to achieve alignment with the infrared image.

[0016] S22. During training, the input visible light image is distorted by bilinear interpolation using a deformation field. The mean square error (MSE) loss between the distorted visible light image and the real data is calculated. The network parameters are optimized using gradient backpropagation, and finally the deformation field converges to a constant state independent of the input.

[0017] Furthermore, S21 includes the following steps:

[0018] S211.PDFNet processes both visible light and infrared inputs simultaneously using a separate feature encoder E. It then calculates local feature similarity in the feature space by computing the inner product between corresponding visible light features and infrared regions. The calculation formula is as follows:

[0019]

[0020] Where corr(i,j) represents the inner product of the feature at position (i,j) in the visible light image and the corresponding position in the infrared image, and C represents the total number of channels. and Let (i,j) and (i+Δi,j+Δj) be the feature vectors at spatial locations (i,j) and (i+Δi,j+Δj) in the visible light and infrared images, respectively, for channel c. The search range within the local D×D region is defined, and the Cat(·) operation concatenates these D×D inner products to form a D×D product at spatial location (i,j). 2 3D feature vector;

[0021] S212. The generated correlation map is input into the subsequent decoder D to predict the deformation field, which indicates the displacement of each pixel in the visible light image.

[0022] Furthermore, the mathematical representation of the distortion of the input visible light image by the applied deformation field in S22 through bilinear interpolation is as follows:

[0023]

[0024] in, This represents a visible light image aligned with an infrared image after being distorted according to a deformation field. p represents the coordinates in the distorted visible image, δp represents the corresponding displacement vector at p in the deformation field, q enumerates all spatial positions in the input visible image, and the bilinear interpolation kernel G(q,p+δp) assigns bilinear weights when q is within a 4-pixel neighborhood of p+δp, otherwise returns zero. Conceptually, this operation interpolates a value in the input visible image and assigns it to position p in the distorted visible image.

[0025] Furthermore, S3 includes the following steps:

[0026] S31. Deploy the trained constant deformation field to the UAV. First, crop the real-time captured high-resolution visible light image to 1920×1024 pixels, then upsample the pre-calculated 640×512 pixel deformation field to 1920×1024 pixels. Each displacement vector in the upsampled field is scaled proportionally to preserve spatial relationships.

[0027] S32. The cropped visible light image is distorted using the scaled deformation field, and finally downsampled to 640×512 pixels to match the resolution of the infrared image, achieving real-time alignment at 33 frames per second.

[0028] Furthermore, the formula for scaling each displacement vector in the upsampled field in S31 to preserve spatial relationships is as follows:

[0029]

[0030] Where, v′ x (i, j) and v′ y (i, j) represents the x and y components of the displacement vector at position (i, j) in the upsampled deformation field, while v x and v y s represents the corresponding component in the original field. x and s y Let s represent the upsampling ratios in the x and y directions, respectively. x =3,s y =2.

[0031] This invention also provides a real-time alignment system for visible and infrared images captured by unmanned aerial vehicles, comprising:

[0032] The preprocessing module is used to crop and downsample the visible light images captured by the drone to match their size to that of the infrared images; and

[0033] The training module, containing the PDFNet network, receives preprocessed visible light and infrared images and trains them to obtain a deformable field that converges to a state independent of the input; and

[0034] Alignment module: This module is used to deploy the constant deformation field obtained from training onto the UAV. After preprocessing the real-time captured visible light image, the deformation field is applied to achieve alignment with the infrared image.

[0035] By adopting the above technical solution, the present invention has the following advantages:

[0036] 1. This invention provides a real-time alignment method for visible light and infrared images taken by UAVs. By introducing a constant deformation field, the alignment problem of visible light and infrared images taken by fixed devices can be solved by directly deforming the visible light image through the deformation field without feature extraction. This enables the alignment of images in any scene, and is applicable to any scenario. It solves the problem of alignment failure in existing technologies on images with cross-modal or low texture.

[0037] 2. This invention obtains a constant deformation field through a training framework, which can be directly deployed onto the UAV. The deformation field can generate high-quality aligned visible images at a rate of 33 frames per second (FPS), achieving real-time alignment of dual-light images on the UAV. This overcomes the shortcomings of existing technologies, which are time-consuming and difficult to implement in real time. It is specifically designed for visible light-infrared image alignment in UAV aerial photography. Compared with general alignment technologies, it has higher professionalism and deployability, and solves the problem of balancing efficiency and accuracy in existing technologies. It is suitable for downstream tasks such as multimodal image fusion and target detection in UAVs. Attached Figure Description

[0038] Figure 1 This is a flowchart of the real-time alignment method for visible and infrared images taken by drones according to the present invention;

[0039] Figure 2 Visible light and infrared images taken by drones, along with their resolution sizes;

[0040] Figure 3 This is a structural diagram of PDFNet of the present invention;

[0041] Figure 4 This is a schematic diagram illustrating how the network finds the optimal solution for aligning the deformation field to gradually stabilize during training. Detailed Implementation

[0042] The technical solution of the present invention will be specifically described below with reference to the accompanying drawings. It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0043] This invention provides a real-time alignment system for visible and infrared images captured by a drone, comprising a preprocessing module, a training module, and an alignment module. The preprocessing module crops and downsamples the visible light image captured by the drone to match its size with the infrared image. The training module includes a PDFNet network, which receives the preprocessed visible light and infrared images and trains them to obtain a deformation field that converges to a state independent of the input. The alignment module deploys the trained constant deformation field to the drone and applies this deformation field to the preprocessed visible light image to achieve alignment with the infrared image.

[0044] This invention also provides a real-time alignment method for visible light and infrared images captured by UAVs. It is the first real-time onboard alignment method for visible light and infrared images with different resolutions and field of view sizes captured by UAVs. A framework specifically designed to solve the deformation field for pixel-level alignment of visible light and infrared images captured by UAVs is described below. Figure 1 As shown, it includes the following steps:

[0045] S1. Preprocessing of visible light and infrared images captured by UAV: ​​Cropping and downsampling the visible light images captured by UAV to match their size with the infrared images;

[0046] S1 includes the following specific steps:

[0047] S11. Preprocessing of UAV aerial images of visible light and infrared: Because UAV aerial images of visible light have higher resolution than infrared images, specifically as follows... Figure 2 As shown, the visible light image collected by the UAV is first preprocessed to obtain a visible light input image that matches the size of the infrared image, which can then be input into the subsequent network model. The visible light image is cropped from 1920×1080 pixels to 1920×1024 pixels, and unnecessary edge areas are removed.

[0048] S12. Then, the pixels of the cropped visible light image are downsampled to 640×512 to match the size of the infrared image taken by the drone. This slight cropping does not result in the loss of valuable information, but rather intentionally removes a small portion of unnecessary edge areas, which can actually improve alignment accuracy to some extent.

[0049] S2. Training and Solving of the Deformation Field for Visible and Infrared Image Alignment: Preprocessed visible and infrared images are input into the designed training scheme to predict the displacement of each pixel within the alignment range of the visible image. Each pixel is rearranged according to its corresponding displacement to achieve alignment between the visible and infrared images. These displacements form the deformation field, which is predicted using PDFNet. The deformation field is obtained through network training and converges to a state independent of the input during training.

[0050] S2 includes the following specific steps:

[0051] S21. The preprocessed visible light image and infrared image are input into the PDFNet network. The PDFNet network processes both inputs simultaneously through a feature encoder, calculates local feature similarity to obtain a correlation map, and then uses a decoder to predict the deformation field. The displacement of each pixel in the deformation field is used to rearrange the pixels of the visible light image to achieve alignment with the infrared image. The PDFNet network structure is an end-to-end network used to predict pixel-level deformation fields, geometrically deforming and aligning the visible light image to the infrared image. Specifically... Figure 3 As shown, it includes a feature encoder and a decoder. The feature encoder extracts features from the visible light image and the infrared image respectively, obtaining their respective feature vectors.

[0052] S21 includes the following steps:

[0053] S211.PDFNet processes both visible and infrared inputs simultaneously using a separate feature encoder (E), and then calculates local feature similarity in the feature space by computing the inner product between corresponding visible and infrared features. This correlation operation can be mathematically expressed as:

[0054]

[0055] Where corr(i,j) represents the inner product of the feature at position (i,j) in the visible light image and the corresponding position in the infrared image, and C represents the total number of channels. and Let (i,j) and (i+Δi,j+Δj) be the feature vectors at spatial locations (i,j) and (i+Δi,j+Δj) in the visible light and infrared images, respectively, for channel c. The search range within the local D×D region is defined, and the Cat(·) operation concatenates these D×D inner products to form a D×D product at spatial location (i,j). 2 3D feature vector;

[0056] S212. The generated correlation map is input into the subsequent decoder (D) to predict the deformation field. This deformation field indicates the displacement of each pixel in the visible light image. For example... Figure 4 As shown, the deformation field specifies the source location of each pixel in the input visible image (white dashed box) and propagates outward.

[0057] S22. During training, the input visible light image is distorted by bilinear interpolation using a deformation field. The mean square error (MSE) loss between the distorted visible light image and the real data is calculated. The network parameters are optimized using gradient backpropagation, and finally the deformation field converges to a constant state that is independent of the input, i.e., a constant deformation field.

[0058] In one specific embodiment, the constant deformation field is then applied to warp the input visible image using bilinear interpolation, mathematically represented as:

[0059]

[0060] in, This represents a visible light image aligned with the infrared image after being warped according to the deformation field. `p` represents the coordinates in the warped visible image, `δp` represents the corresponding displacement vector at position `p` in the deformation field, and `q` enumerates all spatial positions in the input visible image. The bilinear interpolation kernel `G(q,p+δp)` assigns bilinear weights if `q` is within a 4-pixel neighborhood of `p+δp`, otherwise returns zero. Conceptually, this operation interpolates a value in the input visible image and assigns it to position `p` in the warped visible image.

[0061] Finally, the mean squared error (MSE) loss between the distorted visible image and the real data is calculated, and the gradient obtained from this loss is backpropagated through the entire network to train the parameters of PDFNet.

[0062] like Figure 4 As shown, during PDFNet training, the deformation field from different inputs becomes increasingly consistent across different epochs. Simultaneously, the difference between the distorted image and the infrared reference gradually decreases. Ultimately, the field converges to an input-invariant state. The final visualization of the deformation field shows that the distortion originates from a local region (white dashed box) in the input visible image and propagates outwards. This source region essentially corresponds to the content of the infrared image. Crucially, this propagation involves complex non-rigid transformations that go beyond simple scaling or rotation, accurately capturing the geometric distortion between the visible and infrared cameras.

[0063] S3. Constant Deformation Field Deployment for Real-Time Aerial Dual-Light Image Alignment: A constant deformation field is deployed on the UAV to preprocess the real-time captured visible light image, and then the constant deformation field is applied to achieve alignment with the infrared image. This invention, by introducing a constant deformation field, allows the alignment of visible light-infrared images captured by a fixed-configuration UAV to be directly achieved by deforming the visible light image using the constant deformation field, without requiring further input into a network for image alignment.

[0064] S3 includes the following steps:

[0065] S31. Deploy the trained constant deformation field to real-time UAV aerial image alignment. During deployment, since the deformation field is invariant, the pre-computed field can be directly applied to the real-time visible light images captured by the UAV, thus eliminating the need for in-flight network prediction. Specifically, similar to the training phase, the real-time captured high-resolution visible light images are slightly cropped to 1920×1024 pixels. To improve the quality of the distorted images, the pre-computed deformation field (initially 640×512) is upsampled to match the cropped resolution (1920×1024). Subsequently, each displacement vector in the upsampled field is scaled proportionally to preserve spatial relationships:

[0066]

[0067] Where, v′ x (i, j) and v′ y (i, j) represents the x and y components of the displacement vector at position (i, j) in the upsampled deformation field, while v x and v y This represents the corresponding component in the original field. s x and s y These represent the upsampling ratios in the x and y directions, respectively. In this invention, sx =3,s y =2.

[0068] S32. The cropped visible light image is warped using a scaled deformation field to generate a center-aligned visible light image. This result is generated directly from the original high-resolution image, maximizing detail preservation. The warped visible light image is downsampled to 640×512 pixels to match the infrared image resolution, achieving real-time alignment at 33 frames per second. This invention obtains a constant deformation field through a training framework, which can be directly deployed onto the UAV. The deformation field enables the generation of high-quality aligned visible light images at 33 frames per second (FPS), achieving real-time alignment of onboard dual-light images. This is sufficient for downstream tasks including multimodal image fusion and object detection. For a fixed dual-modal sensor, the ideal deformation field remains unchanged within a certain flight altitude of the UAV and the pitch angle of the sensor, perfectly matching our observations. This allows for efficient deployment by directly applying a single pre-computed deformation field to the visible light image, eliminating additional network inference and increasing processing speed to 33 frames per second (FPS), significantly outperforming other existing learnable methods. Comprehensive experiments in image quality, spatial, and information domain consistency validate the superiority of this invention.

[0069] Finally, it should be noted that although the present invention has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Various equivalent changes or substitutions can be made without departing from the concept of the present invention. Therefore, any changes or modifications to the above embodiments within the essential spirit of the present invention will fall within the scope of the claims of the present invention.

Claims

1. A method for real-time alignment of visible light and infrared images taken by unmanned aerial vehicles, characterized in that, Includes the following steps: S1. Preprocessing of visible light and infrared images captured by UAV: ​​Cropping and downsampling the visible light images captured by UAV to match their size with the infrared images; S2. Training and solving the deformation field aligned with visible light and infrared images: Input the preprocessed visible light and infrared images into the PDFNet network, and obtain the deformation field through network training. The deformation field converges to a constant state independent of the input, i.e., a constant deformation field. S2 includes the following steps: S21. The preprocessed visible light image and infrared image are input into the PDFNet network. The PDFNet network processes the two inputs simultaneously through a feature encoder, calculates the local feature similarity to obtain a correlation map, and then the decoder predicts the deformation field. The displacement of each pixel in the deformation field is used to rearrange the pixels of the visible light image to achieve alignment with the infrared image. S22. During training, the input visible light image is distorted by bilinear interpolation using a deformation field. The mean square error (MSE) loss between the distorted visible light image and the real data is calculated. The network parameters are optimized using gradient backpropagation, and finally the deformation field converges to a constant state independent of the input. S3. Constant Deformation Field Deployment for Real-Time Aerial Dual-Light Image Alignment: A constant deformation field is deployed on the UAV to preprocess the real-time captured visible light image, and then the constant deformation field is applied to achieve alignment with the infrared image.

2. The method for real-time alignment of visible and infrared images taken by UAVs according to claim 1, characterized in that, S1 includes the following steps: S11. Preprocess the visible light image collected by the UAV: ​​crop the pixels of the visible light image from 1920×1080 to 1920×1024 and remove unnecessary edge areas; S12. Then, the pixels of the cropped visible light image are downsampled to 640×512 to make its size consistent with the infrared image taken by the drone.

3. The method for real-time alignment of visible and infrared images taken by UAVs according to claim 1, characterized in that, S21 includes the following steps: S211.PDFNet processes both visible light and infrared inputs simultaneously using a separate feature encoder E. It then calculates local feature similarity in the feature space by computing the inner product between corresponding visible light features and infrared regions. The calculation formula is as follows: ； Where corr(i,j) represents the inner product of the feature at position (i,j) in the visible light image and the corresponding position in the infrared image, and C represents the total number of channels. and Let (i,j) and (i+Δi,j+Δj) be the feature vectors at spatial locations (i,j) and (i+Δi,j+Δj) in the visible light and infrared images, respectively, for channel c. The search range within the local D×D region is defined, Cat( The operation connects these D×D inner products to form a product at spatial location (i,j). 3D feature vector; S212. The generated correlation map is input into the subsequent decoder D to predict the deformation field, which indicates the displacement of each pixel in the visible light image.

4. The real-time alignment method for visible and infrared images taken by UAVs according to claim 1, characterized in that, The mathematical representation of the distortion of the input visible light image by the applied deformation field in S22 through bilinear interpolation is as follows: in, This represents a visible light image aligned with an infrared image after being distorted by the deformation field. p Represents the coordinates in the distorted visible light image. δp In the deformed field p The corresponding displacement vector at that location. q Enumerate all spatial locations in the input visible light image, bilinear interpolation kernel G(q,p+δp) exist q lie in p+δp If the value is within a 4-pixel neighborhood, a bilinear weight is assigned; otherwise, zero is returned. This operation interpolates a value in the input visible light image and assigns it to the location in the distorted visible light image. p .

5. A method for real-time alignment of visible and infrared images taken by UAVs according to claim 1, characterized in that, S3 includes the following steps: S31. Deploy the constant deformation field obtained from training to the UAV. First, crop the high-resolution visible light image captured in real time to 1920×1024 pixels, and then upsample the pre-calculated 640×512 pixel deformation field to 1920×1024 pixels. Each displacement vector in the upsampled field is scaled proportionally to preserve spatial relationships. S32. The cropped visible light image is distorted using the scaled deformation field, and finally downsampled to 640×512 pixels to match the resolution of the infrared image, achieving real-time alignment at 33 frames per second.

6. A method for real-time alignment of visible and infrared images taken by UAVs according to claim 5, characterized in that, The formula for scaling each displacement vector in the upsampled field in S31 to preserve spatial relationships is as follows: in, and Indicates the position in the upsampled deformation field ( i , j The x and y components of the displacement vector at position ) and and This represents the corresponding component in the original field. and These represent the upsampling ratios in the x and y directions, respectively. , .

7. A real-time alignment system for visible and infrared images captured by unmanned aerial vehicles, characterized in that, The method for real-time alignment of visible and infrared images taken by UAVs as described in any one of claims 1-6 includes: The preprocessing module is used to crop and downsample the visible light images captured by the drone to match their size with the infrared images; The training module, which includes the PDFNet network, is used to receive preprocessed visible light and infrared images and train them to obtain a deformable field that converges to a state independent of the input. The alignment module is used to deploy the trained constant deformation field to the UAV. After preprocessing the real-time captured visible light image, the deformation field is applied to achieve alignment with the infrared image.

Images