A method for fast registration of visible and infrared images of a drone

Abstract

This invention discloses a rapid registration method for visible light and infrared images from a UAV, belonging to the fields of image processing and UAV remote sensing technology. The method includes: acquiring camera intrinsic parameters, calculating the intrinsic parameter ratio, and obtaining a fixed offset value through benchmark calibration in a high-altitude scene; simultaneously acquiring visible light and infrared images of the same scene, and obtaining the flight altitude and the physical baseline distance between the two cameras; scaling the infrared image according to the camera intrinsic parameters to determine the registration window size; constructing a physical offset model to establish the total offset value of the registration window center relative to the visible light image center; determining the registration window center by applying the total offset value to the visible light image center in the visible light image coordinate system, and calculating the coordinates of the four corner points according to the registration window size to achieve accurate registration. This method features high computational efficiency, fast registration speed, and no need for a large training dataset, thus meeting the rapid registration requirements of visible light and infrared images from UAVs.

Description

Technical Field

[0001] This invention relates to the fields of image processing and UAV remote sensing, and particularly to a rapid registration method for visible light images and infrared images from a UAV. Background Technology

[0002] With the rapid development of UAV remote sensing, power line inspection, and military reconnaissance technologies, multimodal image sensing technology has been widely applied. Visible light images, with their high resolution, high definition, and rich texture details, provide intuitive visual information; while infrared images, by sensing differences in the thermal radiation of objects, possess strong capabilities to penetrate fog and smoke, as well as target detection in low-light environments. Registering and fusing visible light and infrared images from the same scene not only overcomes the limitations of a single sensor but also significantly enhances the accuracy of target recognition and the system's environmental adaptability.

[0003] However, in practical applications, the imaging principles of visible light and infrared sensors are fundamentally different, and their intrinsic parameters and resolutions differ significantly, making image registration between the acquired images challenging. Existing image registration techniques can be mainly categorized as follows:

[0004] 1) Feature-based registration methods: These methods extract feature points using operators such as SIFT, SURF, or ORB. While these methods perform well in single-modal scenarios, in cross-modal tasks, the inherent defects of infrared images, such as low signal-to-noise ratio and lack of texture details, result in insufficient discriminative power of feature descriptors and poor robustness of feature point pairing, making it difficult to construct high-precision homography matrices.

[0005] 2) Deep learning-based registration methods: These methods utilize architectures such as CNNs or GANs to extract semantic features. While capable of handling complex nonlinear distortions, they are highly dependent on large-scale, high-quality labeled datasets. In specific industry applications or small-sample scenarios, the limited generalization ability of the model and the high training costs make it difficult to achieve plug-and-play rapid deployment.

[0006] 3) Region-based registration methods: Matching is performed by calculating the similarity index of local windows. Because this type of method is extremely sensitive to gray-level heterogeneity between cross-modal images, when processing visible light and infrared images with significant gray-level differences and large contrast polarities, the similarity metric struggles to establish a stable mapping relationship between heterogeneous data. This leads to a significant degradation in the computational accuracy of the matching operator, and may even result in registration failure due to the inability to capture the global optimal solution.

[0007] Especially in drone-borne operations, real-time changes in flight altitude directly cause dynamic changes in the relative parallax between two cameras. Traditional registration methods mostly ignore this physical relationship or cannot perform fast registration without relying on complex feature calculations. Therefore, how to construct a computationally compact registration scheme that does not require large-scale feature matching and can correct displacement deviations in real time based on flight status is a key problem that urgently needs to be solved in the field of multimodal image processing. Summary of the Invention

[0008] The problem to be solved by this invention is to provide a fast registration method for visible light images and infrared images from UAVs. The aim is to solve the bottleneck problems of traditional feature matching algorithms, such as large computational load, large training dataset requirements, and poor registration robustness, by constructing a physical offset model based on UAV altitude feedback, while ensuring registration accuracy. This achieves efficient alignment of visible light images and infrared images.

[0009] This invention adopts the following technical solution: a rapid registration method for visible light images and infrared images from a UAV, comprising the following steps:

[0010] S1. Two-way fixed offset value calculation: Obtain camera intrinsic parameters, calculate intrinsic parameter ratio, and calculate fixed offset value in high-altitude scenes;

[0011] S2. Data Acquisition and Parameter Extraction: Simultaneously acquire visible light and infrared images of the same scene, and obtain the drone's flight altitude. and the physical baseline distance between the two camera lenses ;

[0012] S3. Registration window size calculation: The infrared image is scaled according to the internal reference ratio to determine the registration window size;

[0013] S4. Construct a physical offset model: Calculate the dynamic offset value, and based on the fixed offset value and the dynamic offset value, calculate the total offset value of the center of the two images in the horizontal and vertical directions respectively;

[0014] S5. Registration window coordinate calculation: The total offset value is used to determine the center of the registration window in the visible light image coordinate system, and the coordinates of the four corner points are calculated to complete the image registration.

[0015] Specifically, the camera intrinsic parameter is the pixel focal length of the visible light camera. Infrared camera pixel focal length The physical baseline distance between the two camera lenses Based on the relative mounting position of the camera, it is specifically decoupled into a horizontal baseline component in the image coordinate system. and vertical baseline components .

[0016] Preferably, in step S1, benchmark calibration is performed, including the following sub-steps:

[0017] S1.1, Based on the pixel focal length of the visible light camera With infrared camera pixel focal length Calculate internal reference ratio The infrared image is resampled and scaled to align the visible light image with the infrared image in terms of scale.

[0018] S1.2. Select the center of the visible light image as a coordinate reference, and align the center of the infrared image with it to complete the initial positioning. Then, compensate for and eliminate the ghosting of the same feature points between the two images by translating the infrared image. Record the horizontal and vertical displacement pixel values ​​of the infrared image center relative to the initial position when the images reach the overlap state. By averaging multiple sets of images, the final horizontal fixed offset value is determined. and vertical fixed offset value .

[0019] Preferably, in step S3, the registration window size is calculated, and the specific process is as follows:

[0020] Calculated based on S1 The infrared image is scaled and rounded to determine the size of the effective registration area on the visible light image, i.e., the registration window size.

[0021] Preferably, in step S4, the physical offset model is constructed by performing the following sub-steps:

[0022] S4.1 Calculation of dynamic offset value: using the formula and Calculate the horizontal and vertical dynamic offset values ​​as a function of height, respectively;

[0023] S4.2 Total Offset Calculation: The dynamic offset value is superimposed with the fixed offset value to obtain the total horizontal offset value. and total vertical offset value .

[0024] Preferably, the specific process of step S5 is as follows:

[0025] Centered on visible light image Using the total offset value as a reference, the coordinates of the four corner points are calculated according to the registration window size. The infrared image is then scaled to the range of the registration window to complete the registration.

[0026] The present invention also provides: an electronic device, comprising:

[0027] One or more processors;

[0028] A storage device on which one or more programs are stored;

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the fast registration methods described above.

[0030] The present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in any of the above-mentioned fast registration methods.

[0031] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0032] This invention designs a general, efficient, and interference-resistant registration method based on physical geometric constraints, avoiding complex feature descriptor extraction and matching operations, and greatly improving the real-time performance of multi-source image alignment. By introducing a fixed offset term for high-altitude calibration, it effectively alleviates the offset caused by hardware installation errors, while dynamic disparity compensation enhances the model's adaptability at different flight altitudes. It balances registration accuracy and computational resource consumption, does not rely on large-scale sample data for offline training, greatly reduces the algorithm's dependence on specific datasets, has stronger scene generalization ability and environmental adaptability, and achieves stable and efficient visual alignment representation. Attached Figure Description

[0033] Figure 1 This is a flowchart of the rapid registration method of the present invention;

[0034] Figure 2 This is a schematic diagram of the camera used in an embodiment of the present invention;

[0035] Figure 3 This is a comparison image of a set of visible light images and infrared images in a high-altitude scene before and after the correction of fixed offset values, according to an embodiment of the present invention.

[0036] Figure 4 This is an example scenario diagram used in embodiments of the present invention;

[0037] Figure 5 This is a diagram illustrating the principle of parallax.

[0038] Figure 6 This is a registration effect diagram for an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0040] In one embodiment of the present invention, a rapid registration method for visible light images and infrared images from a UAV is provided, the process of which is as follows: Figure 1 As shown.

[0041] This embodiment uses an outdoor shooting and laboratory registration environment, and the camera used is as follows: Figure 2 As shown, the experimental data consists of visible light and infrared images captured by the wide-angle and infrared cameras of a DJI Mavic 3T Enterprise series drone. The visible light camera has a focal length of 2850 pixels, and the infrared camera has a focal length of 734 pixels. The physical baseline distance between the two lenses is... 2cm The altitude was 0 cm. The visible light image resolution was 4000×3000, and the infrared image resolution was 640×512. Five pairs of visible light and infrared images were captured for calibration, and 275 pairs of visible light and infrared images were captured for registration. In addition, the flight altitude was extracted from the EXIF ​​information of the UAV image.

[0042] Based on the experimental data, the registration window was determined, and the specific steps are as follows:

[0043] S1. Calculate the fixed offset value:

[0044] S1.1. Calculate the internal reference ratio of 3.883 based on the pixel focal length of the visible light camera and the pixel focal length of the infrared camera. Then, resample and scale the infrared image to align the scale of the visible light image and the infrared image.

[0045] S1.2 In this embodiment, five sets of different scene images with a flight altitude greater than 70m are selected for calibration calculation.

[0046] Using the visible light image center as a reference, ghosting of corresponding feature points is eliminated by translating the infrared image, and the horizontal and vertical displacement pixel values ​​of the infrared image center when each group of images reaches the overlapping state are recorded. The above 5 groups of data are averaged, and the final fixed horizontal offset value is determined to be -60 pixels, and the fixed vertical offset value is 0 pixels.

[0047] One set of scene images shows a comparison of the effects before and after offset compensation, as follows: Figure 3 As shown, Figure 3 (a) in the image shows the effect before the offset. Figure 3 (b) in the diagram shows the effect after offsetting.

[0048] S2. Scale the infrared image according to the internal reference ratio, round it, and determine the registration window size as 2485×1988;

[0049] S3. Construct the physical offset model. The specific steps are as follows:

[0050] S3.1 In this embodiment, a pair of images is selected as an example, such as... Figure 4 As shown in (a) and (b), the height is 2 m. Based on the parallax principle, the dynamic offset value is calculated, and the horizontal dynamic offset value is 28.5 pixels and the vertical dynamic offset value is 0 pixels.

[0051] Among them, the parallax principle is as follows: Figure 5 As shown, based on plane geometry and trigonometric formulas, the following relationships exist:

[0052] ;

[0053] in, This refers to the camera's focal length (pixel focal length). The baseline length between the two cameras. The depth of the object from the camera; and These represent the x-coordinates (or pixel coordinates) of the target point in the left and right images, respectively. The difference between the two is the disparity. .

[0054] S3.2 Calculate the total offset value: the horizontal offset value is -31.5 pixels, and the vertical offset value is 0 pixels;

[0055] S4, Centered on the visible light image After superimposing the total offset value, the coordinates of the four corner points of the registration window are obtained from the registration window size, as follows: , , , The infrared image is scaled to this range to complete the registration, and the result is as follows. Figure 6 As shown.

[0056] As can be seen, the method of the present invention can effectively solve the problem that feature point matching is prone to failure in weak texture scenes due to the large difference between infrared and visible light image bands, as well as the parallax problem caused by the changes in installation position and height of different sensors. It also has the characteristics of high computational efficiency, fast registration speed, and no need for a large training dataset, and can meet the rapid registration requirements of UAV visible light images and infrared images.

[0057] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for rapid registration of visible light images and infrared images from a UAV, characterized in that, Includes the following steps: S1. Obtain camera intrinsic parameters, calculate the intrinsic parameter ratio, and calculate a fixed offset value in high-altitude scenes: S2. Simultaneously acquire visible light and infrared images of the same scene, and obtain the drone's flight altitude and the physical baseline distance between the two camera lenses; S3. Scale the infrared image according to the internal reference ratio to determine the registration window size; S4. Construct a physical offset model and calculate the total offset values ​​of the registration window center relative to the visible light image center in the horizontal and vertical directions, respectively. S5. Apply the total offset value to the visible light center to determine the center of the registration window, and calculate the coordinates of the four corner points to achieve image registration.

2. The camera intrinsic parameters include: visible light camera pixel focal length. Infrared camera pixel focal length ; The physical baseline distance between the two camera lenses is interpreted as a horizontal baseline component and a vertical baseline component in the image coordinate system, depending on the relative installation positions of the two cameras.

3. The rapid registration method according to claim 1, characterized in that, The calculation of the fixed offset value in step S1 includes the following sub-steps: S1.1 Acquire visible light and infrared images of high-altitude scenes, based on the pixel focal length of the visible light camera. With infrared camera pixel focal length Proportional relationship, calculation of internal reference ratio ;according to The infrared image is resampled and scaled to align the visible light image with the infrared image in terms of scale; the internal reference ratio is calculated using the following formula: ； S1.2 Using the visible light image center as a reference, align the infrared image center with it, and eliminate the ghosting of corresponding feature points in the two images by translating the infrared image; obtain the horizontal and vertical displacement values ​​of the infrared image center relative to the initial position when the two images reach the overlap state, and determine the fixed horizontal offset by averaging the values ​​after calculating multiple sets of images. and vertical fixed offset .

4. The rapid registration method according to claim 1, characterized in that, The scaling described in step S3 specifically refers to: Based on the internal reference ratio obtained in step S1 The infrared image is scaled and rounded to determine the effective registration area size on the visible light image, which is then used as the registration window size.

5. The rapid registration method according to claim 3, characterized in that, Step S4 involves constructing the physical offset model, which includes the following sub-steps: S4.1 Calculate the dynamic offset value: Calculate the dynamic offset value as a function of height in the horizontal and vertical directions respectively, based on the parallax principle. and The calculation formula is as follows: ； ； in, and These are the horizontal baseline component and the vertical baseline component, respectively. Flight altitude; S4.2 Calculate the total offset value: total horizontal offset value and total vertical offset value The calculation formula is as follows: ； ； in, A fixed horizontal offset value. This is a fixed vertical offset value. This is the horizontal dynamic offset value. This is the vertical dynamic offset value.

6. The rapid registration method according to claim 4, characterized in that, The application of the total offset value to achieve registration in step S5 is specifically as follows: In the visible light image coordinate system, with the center point of the visible light image as the reference point... Based on, superimposed and Determine the center of the registration window; Calculate the coordinates of its four corner points, scale the infrared image to the range of the registration window, and complete the registration.

7. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the fast registration method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the rapid registration method according to any one of claims 1 to 6.

Images