Method for registering and fusing infrared images and SAR images during dynamic flight of unmanned aerial vehicles

Abstract

The present invention relates to the technical field of image registration of UAV payload systems, and particularly to a method for registering and fusing infrared images and SAR images during the dynamic flight of a UAV. It includes measuring and calibrating the relationship between the focal length of the infrared camera and the infrared field of view angle in the Y direction on the ground, recording the relationship between the field of view angle and the focal length as a file, and storing it on the camera control board; the camera control board controls the movement of the infrared camera through the PID algorithm until the field of view angle of a single pixel of the infrared image matches the field of view angle of a single pixel of the SAR image; extracting and matching the heterogeneous image descriptors, storing the projection relationship of each pixel coordinate as a matrix C, and calling to complete the registration; calculating the local contrast and information entropy of the infrared image and the SAR image, dynamically adjusting the fusion ratio, and generating the registered and fused image. The advantages are as follows: only simple calibration is required on the ground; after changing the focal length, the registration algorithm is run on the airborne computer, and the multi-scale descriptors are combined to improve the efficiency and accuracy of feature matching.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration of UAV payload systems, and particularly to a method for registering and fusing infrared images and SAR images during the dynamic flight of a UAV. Background Art

[0002] Traditional image registration methods usually require complex ground calibration processes, which include precise measurements of the offset of the optical axis centers of infrared and SAR radars, edge distortion, and pixel mapping relationships. These calibration processes are often time-consuming and laborious, and are only applicable to the case of fixed focal length lenses. However, in the integrated payload system of UAV airborne optoelectronic radars, the infrared is equipped with a variable focal length lens, and the ground resolution of the SAR radar can usually be adjusted, which makes the traditional calibration methods no longer applicable. In addition, due to the dynamic nature of the UAV platform and the real-time requirements of the mission, traditional image registration methods face challenges in practical applications. When the UAV is performing monitoring tasks, changes in the camera focal length are normal, which requires the image registration algorithm to be able to quickly adapt to changes in the infrared focal length and the ground resolution transformation of the SAR radar to achieve real-time processing.

[0003] In the prior art, the registration of infrared images and SAR images usually requires the execution of cumbersome ground calibration procedures, including calibrating the offset of the optical axis centers of infrared and SAR, imaging mode differences, pixel mapping relationships, etc. The calibration process is complex and only applicable to the case where the infrared is a fixed focal length lens and the SAR is a fixed parameter configuration. The infrared camera of the integrated optoelectronic radar payload has a variable focal length, and the SAR system has a variable parameter configuration. The workload of calibrating the center offset, imaging mode adaptation, pixel mapping, etc. for each focal length or parameter configuration is huge.

[0004] The main limitations of the prior art are its complex ground calibration process and the lack of an image registration solution applicable to variable focal length lenses; the lack of an algorithm that can adapt to the focal length changes of the variable focal length infrared camera and the variable ground resolution SAR radar of the UAV airborne optoelectronic radar payload and achieve fast and accurate registration. Summary of the Invention

[0005] In order to solve the above problems, the present invention provides a method for registering and fusing infrared images and SAR images during the dynamic flight of a UAV.

[0006] The object of the present invention is to provide a method for registering and fusing infrared images and SAR images during the dynamic flight of a UAV, which specifically includes the following steps:

[0007] S1. Ground calibration: Measure and calibrate the relationship between the focal length of the infrared camera and the infrared field of view angle in the Y direction on the ground, record the relationship between the field of view angle and the focal length as a file, and store it on the camera control board;

[0008] S2. Coarse matching of ground pixel resolution: The camera control board controls the movement of the infrared camera through the PID algorithm until the field of view angle of a single pixel in the infrared image matches the field of view angle of a single pixel in the SAR image;

[0009] S3. Extraction and registration of image descriptors: Extract the heterogeneous image descriptors of the infrared image and the SAR image, match the descriptors, and store the projection relationship of each pixel coordinate of the infrared image and the SAR image as matrix C; Call matrix C to complete the registration;

[0010] S4. Image fusion: According to the registered images, convert the infrared image format from RGB to YUV, overlay the corresponding brightness values of the SAR image on the Y channel, calculate the local contrast and information entropy of the infrared image and the SAR image, dynamically adjust the fusion ratio of the infrared image and the SAR image, and generate the registered and fused image.

[0011] Preferably, the measurement and calibration method in step S1 specifically includes: using a folding infrared SAR dual-optical co-field-of-view light pipe to gradually measure the field of view angle of the infrared camera in the Y direction; The folding infrared SAR dual-optical co-field-of-view light pipe contains a crosshair, and the stepping length is 0.1 m.

[0012] Preferably, the method for adjusting the field of view angle of a single pixel in the infrared image to match the field of view angle of a single pixel in the SAR image in step S2 is specifically as follows:

[0013] The airborne computer calculates the focal length of the infrared camera when the resolution of each pixel of the infrared camera is equal to the ground pixel resolution of the SAR image according to the flight altitude of the UAV and the azimuth and pitch angles of the optoelectronic payload ; The camera control board controls the infrared camera to adjust to the focal length until the field of view angle of a single infrared pixel matches the field of view angle of a single SAR pixel.

[0014] Preferably, the heterogeneous image descriptors of the infrared image and the SAR image in step S3 include phase consistency descriptors and gradient descriptors; The method of weighted correlation distance is used for matching; The specific method is as follows:

[0015] S301. Calculate the phase consistency descriptor of the heterogeneous image to measure the phase of specific frequency components in the heterogeneous image; For each pixel, the phase can be calculated by the following formula:

[0016] ;

[0017] where is the pixel coordinate, w is different scales, is the phase at scale w, and M represents the number of sizes;

[0018] S302. Calculate the gradient descriptors of the heterogeneous images;

[0019] The gradient direction descriptors of the heterogeneous images are as follows:

[0020] ; ;

[0021] Among them, and are the gradients in the x - direction and y - direction at the scale of m respectively, and 𝐼 is the brightness of the image. Thus, the gradient descriptor of the heterogeneous image is expressed as:

[0022]

[0023] ;

[0024] S303. Set the statistical scale as m. Taking (x, y) pixels as the center, the gradient histogram of the surrounding 10×10 pixels is expressed as follows:

[0025] ;

[0026] S304. Based on the target size of the image, take M / 2 scales upward and M / 2 scales downward, a total of M scales; ; The phase - consistency descriptor , the gradient descriptor , and the gradient histogram constitute the descriptor vector at the m - th scale:

[0027] ;

[0028] S305. Use the method of weighted correlation distance for matching; the expression is as follows:

[0029] ;

[0030] In the formula, WCD represents the weighted correlation distance, that is, the weighted sum of the differences between the descriptor vectors of the infrared image and the SAR image from scale 1 to scale M; m is the scale of the descriptor; M represents the number of sizes; and respectively represent the multi - scale descriptor vectors of each feature point in the infrared image and the SAR image; is the weight at the m - th scale, expressed as: ; is the variance of the descriptor vectors and at the m - th scale;

[0031] When the WCD value is greater than 0.5, it is considered that and the matching is successful; the projection relationship of each pixel coordinate between the infrared image and the SAR image is stored as matrix C.

[0032] Preferably, step S4 specifically includes the following sub-steps:

[0033] S401. Calculate the local contrast of the infrared image and the SAR image; normalize the local contrast of the infrared image and the SAR image to obtain the weights of the local contrast of the infrared image and the SAR image;

[0034] S402. Calculate the information entropy of the infrared image and the SAR image, normalize the information entropy of the infrared image and the SAR image to obtain the weights of the information entropy of the infrared image and the SAR image;

[0035] S403. Convert the RGB format file of the infrared image to YUV, and according to matrix C, make the Y-channel brightness value Y ir of the infrared image Y SAR superimpose with the Y-channel brightness value Y of the SAR image and perform pixel-by-pixel fusion to generate the registered and fused image

[0036] ;

[0037] In the formula, Y fused represents the registered and fused image, Y ir represents the Y-channel brightness value of the infrared image, Y SAR is the Y-channel brightness value of the SAR image; represents the weight of the infrared image based on the local contrast, represents the weight of the SAR image based on the local contrast; represents the weight of the infrared image based on the information entropy, represents the weight of the SAR image based on the information entropy.

[0038] Preferably, step S401 specifically includes the following sub-steps:

[0039] S4011. Determine the local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel values in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast;

[0040] The local standard deviation calculation formulas of the infrared image and the SAR image are as follows:

[0041] ;

[0042] ;

[0043] Wherein, is the pixel value of the infrared image at the coordinate ( x , y ); is the pixel value of the SAR image at the coordinate ( x , y ); is the average pixel value of the local region Ω f in the infrared image or the SAR image, |Ω f | represents the total number of pixels in the local region Ω f in the infrared image or the SAR image;

[0044] The local contrast expression of the infrared image or the SAR image is as follows:

[0045] ;

[0046] ;

[0047] S4012. Normalize the local contrasts of the infrared image and the SAR image so that the sum of IRLocalContrast and SARLocalContrast is 1; obtain the weights of the local contrasts of the infrared image and the SAR image:

[0048] ;

[0049] Wherein, represents the weight of the infrared image based on the local contrast, represents the weight of the SAR image based on the local contrast.

[0050] Preferably, step S402 specifically includes the following sub-steps:

[0051] S4021. Use a histogram to represent the number of times the gray value 𝑖 appears in the infrared image or the SAR image; for the gray value of each pixel point in the gray image or , perform the following operations:

[0052] ;

[0053] Wherein, and represent the infrared histogram and the SAR histogram; i represents the gray value; represents the indicator function; when is equal to i, is 1, otherwise it is 0; when When it is equal to i, it is 1, otherwise it is 0; represents the gray value of the pixel at coordinates (x, y) in the infrared image, represents the gray value of the pixel at coordinates (x, y) in the SAR image;

[0054] S4022. Perform normalization processing to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image:

[0055] ;

[0056] ;

[0057] In the formula, and are the probability distributions of each gray value of the infrared image and the SAR image respectively, and are the width and height of the infrared image respectively, and are the width and height of the SAR image respectively;

[0058] S4023. Use the Shannon information entropy formula to multiply the probability of each gray value by its logarithm to the base 2 and sum over all 256 possible gray values to calculate the infrared image information entropy 𝐻ir and the SAR image information entropy 𝐻SAR; the calculation formula is as follows:

[0059] ;

[0060] ;

[0061] S4024. Normalize the information entropy of the infrared image and the SAR image so that H ir and H SAR sum to 1 to obtain the weights of the information entropy of the infrared image and the SAR image:

[0062] ;

[0063] In the formula, represents the weight of the infrared image based on information entropy, represents the weight of the SAR image based on information entropy.

[0064] Preferably, the value range of the information entropy is 0 to 8 bits; when all pixels have the same gray value, H ir = 0, H SAR = 0; when the probability of each gray value appearing is equal, the information entropy H ir = 8,H SAR = 8.

[0065] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0066] The present invention only needs to simply calibrate the field of view angles of infrared and SAR on the ground. After zooming, on the airborne computer, run the registration algorithm, and combine multi-scale descriptors and optimized search strategies to improve the efficiency and accuracy of feature matching. It is applicable to the registration of infrared SAR images under different resolutions and when the infrared SAR radar is a zoom lens, especially applicable to the registration of images under the variable focal length infrared camera and variable ground resolution SAR radar of the integrated payload of the unmanned aerial vehicle (UAV) airborne optoelectronic radar, etc. It has broad application prospects and practical application value, and significantly reduces the workload of registration calibration. Description of the Drawings

[0067] Figure 1 is a flowchart of a method for registering and fusing infrared images and SAR images during the dynamic flight of a UAV according to an embodiment of the present invention.

[0068] Figure 2 is an infrared image taken by a UAV aerial photography according to an embodiment of the present invention.

[0069] Figure 3 is a SAR image taken by a UAV aerial photography according to an embodiment of the present invention.

[0070] Figure 4 is a fused registration image generated after fusing the infrared image and the SAR image on the UAV according to an embodiment of the present invention. Detailed Embodiments

[0071] In the following, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are denoted by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, their detailed descriptions will not be repeated.

[0072] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

[0073] The present invention aims to solve the limitations of the prior art and provide a new image registration method. This method is applicable to the integrated payload system of the UAV airborne optoelectronic radar, can reduce the calibration workload, and improve the efficiency and accuracy of image registration. By simply calibrating the field of view angle on the ground and combining multi-scale descriptors and optimized search strategies, the registration algorithm can be run in real time on the airborne computer to meet the requirements of real-time image processing of the UAV.

[0074] See Figure 1 , the present invention provides a method for registering and fusing infrared images and SAR images during the dynamic flight of an unmanned aerial vehicle, specifically including the following steps:

[0075] S1. Ground calibration: Install the infrared camera on the airborne optoelectronic turret of the unmanned aerial vehicle; Measure and calibrate the relationship between the focal length of the infrared camera and the infrared Y-direction field of view angle on the ground, record the relationship between the field of view angle and the focal length as a file, and store it on the camera control board;

[0076] The measurement and calibration method specifically includes: Use a folding infrared SAR dual-optical co-field-of-view light pipe (the light pipe contains a crosshair, and the stepping length is 0.1 m) to gradually measure the infrared camera's Y-direction field of view angle;

[0077] Specifically, the optoelectronic turret is internally equipped with a measured azimuth and elevation encoder; Assume that the focal length of the infrared camera lens at this time is IR_F, lock the elevation angle of the optoelectronic turret at 0, align the upper edge of the infrared camera with the crosshair of the light pipe, and read the azimuth angle IRup of the optoelectronic turret at this time; Rotate the optoelectronic turret so that the lower edge of the infrared camera aligns with the crosshair of the light pipe, and read the azimuth angle IRdown of the optoelectronic turret at this time. Then, at the infrared camera focal length IR_F, the infrared Y-direction field of view angle is IR_Y = IRup - IRdown; Record the corresponding relationship between the infrared lens focal length and the field of view angle into the array c.

[0078] S2. Coarse matching of ground pixel resolution: The camera control board controls the movement of the infrared camera through the PID algorithm until the field of view angle of a single pixel of the infrared image matches the field of view angle of a single pixel of the SAR image;

[0079] The specific operation is as follows: The airborne computer calculates the focal length of the infrared camera when each pixel of the infrared camera is equal to the ground pixel resolution of the SAR image according to the flight altitude of the unmanned aerial vehicle and the azimuth and elevation angles of the optoelectronic payload ; The camera control board controls the infrared camera to adjust to the focal length until the field of view angle of a single infrared pixel matches the field of view angle of a single SAR pixel.

[0080] Each pixel of the infrared image and the SAR image has the same ground pixel resolution. Then, the derivation of the infrared lens focal length relationship is as follows: If the ground pixel resolution of the SAR image selected by the ground station is 0.2 m, it means that the ground size represented by each pixel of the SAR is 0.2 m × 0.2 m. The airborne computer calculates the focal length of the infrared camera when each pixel of the infrared camera represents a ground size of 0.2 m × 0.2 m according to the flight altitude of the unmanned aerial vehicle and the azimuth and elevation angles of the optoelectronic payload;

[0081] Considering the elevation angle θ The calculation formula for the focal length of the infrared camera is:

[0082] ;

[0083] Wherein, f is the focal length of the infrared camera; H is the flight altitude of the UAV; θ is the pitch angle of the optoelectronic payload, i.e., the angle between the camera and the horizontal plane; S is the size of the pixels of the infrared camera; P is the ground size represented by the pixel, which is 0.2 meters here.

[0084] Similarly, calculate the focal length of the infrared camera when the SAR is at other resolutions; the camera control board controls the infrared to adjust to the corresponding focal length through the PID algorithm , and adjust repeatedly until the field of view angle of a single pixel of the infrared matches the field of view angle of a single pixel of the SAR.

[0085] S3. Image descriptor extraction and registration: Extract the heterogeneous image descriptors of the infrared image and the SAR image, match the descriptors, and store the projection relationship of each pixel coordinate of the infrared image and the SAR image as matrix C; call matrix C to complete the registration;

[0086] The heterogeneous image descriptors of the infrared image and the SAR image include phase consistency descriptors and gradient descriptors; the method of matching the descriptors is to combine the phase consistency descriptors and the gradient descriptors, extract the general feature descriptors of the infrared image and the SAR image, and use the method of weighted correlation distance (WCD) for matching; the specific method is as follows:

[0087] S301. Calculate the phase consistency descriptor of the heterogeneous image to measure the phase of specific frequency components in the heterogeneous image; for each pixel, the phase can be calculated by the following formula:

[0088] ;

[0089] Wherein, is the pixel coordinate, w is different scales, is the phase at scale w, and M represents the number of sizes (usually taking the value of 10);

[0090] S302. Calculate the gradient descriptor of the heterogeneous image;

[0091] The gradient direction descriptor of the heterogeneous image is as follows:

[0092] ; ;

[0093] Wherein, and are the gradients in the x - direction and y - direction at the scale of m respectively, and 𝐼 is the brightness of the image. Thus, the gradient descriptor of the heterogeneous image is expressed as:

[0094]

[0095] ;

[0096] S303. Set the statistical scale as m. The gradient histogram of 10×10 pixels around the center of the (x, y) pixel is expressed as follows:

[0097] ;

[0098] S304. Based on the target size of the image, take M / 2 scales upward and M / 2 scales downward, for a total of M scales. The phase - consistency descriptor at the m - th scale , the gradient descriptor (gradient magnitude) , and the gradient histogram constitute the descriptor vector at the m - th scale:

[0099] ;

[0100] Both the infrared image and the SAR image are operated according to the above - mentioned method. Each feature point in the obtained infrared image and SAR image generates a rich multi - scale descriptor vector and , which effectively describes the local structure information of the infrared image and the SAR image and provides a solid foundation for feature matching.

[0101] S305. Use the method of weighted correlation distance (WCD) for matching; the expression is as follows:

[0102] ;

[0103] In the formula, WCD represents the weighted correlation distance, that is, the weighted sum of the differences between the descriptor vectors of the infrared image and the SAR image from scale 1 to scale M; m is the scale of the descriptor (the value range of m is from 1 to M); M represents the number of sizes (usually taken as 10); and respectively represent the multi - scale descriptor vectors of each feature point in the infrared image and the SAR image; is the weight at the m - th scale, expressed as: ; is the descriptor vector of the m-th scale and variance;

[0104] When the WCD value is greater than 0.5, it is considered that and the matching is successful; store the projection relationship of each pixel coordinate in the infrared image and the SAR image as matrix C.

[0105] Principle brief: After step S2, in theory, it has been ensured that the infrared ground pixel resolution is basically the same as the SAR ground pixel resolution. However, due to the field of view angle calibration error in step S1 and the focal length control error in step S2, there are slight differences between the infrared ground pixel resolution and the SAR ground pixel resolution. Therefore, a registration operation is required. The calculation of the common feature descriptor of the infrared image and the SAR image is a key step in the matching of the infrared image and the SAR image, which provides a unique vector representation for each feature point of the infrared image and the SAR image.

[0106] The weighted correlation distance (WCD) method not only considers the Euclidean distance between feature vectors but also considers the correlation between feature dimensions. This method is particularly suitable for situations where there are internal connections between feature dimensions.

[0107] Because the infrared SAR uses an external trigger mode to ensure that the infrared SAR is exposed at the same moment. After exposure, a new frame of infrared image and SAR image arrive at the airborne computer. Just by calling matrix C, the real-time registration of the infrared SAR can be completed, effectively guaranteeing the real-time performance of the registration.

[0108] S4. Image fusion: According to the registered images, convert the infrared image format from RGB to YUV, and overlay the corresponding brightness value of the SAR image on the Y channel, calculate the local contrast and information entropy of the infrared image and the SAR image, and dynamically adjust the fusion ratio of the infrared image and the SAR image to generate a registered fusion image; specifically, it includes the following sub-steps:

[0109] S401. Calculate the local contrast of the infrared image and the SAR image; normalize the local contrast of the infrared image and the SAR image to obtain the weights of the local contrast of the infrared image and the SAR image; specifically, it includes the following sub-steps:

[0110] S4011. Determine the local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel values in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast.

[0111] Local standard deviation is the local area Measure of the fluctuation of internal pixel values. The calculation formulas for the local standard deviation of infrared images and SAR images are as follows:

[0112] ;

[0113] ;

[0114] In the formula, is the pixel value of the infrared image at the coordinate ( x , y ), is the pixel value of the SAR image at the coordinate ( x , y ), is the average pixel value of the local region Ω f in the infrared image or SAR image, and ∣Ω f ∣ represents the total number of pixels in the local region Ω f in the infrared image or SAR image.

[0115] The local standard deviation is obtained by calculating the sum of the squares of the deviations of the pixel values within the local region Ω f from their average value μf , and then taking the square root. This value reflects the degree of dispersion of pixel intensities within the feature region and is an indicator for measuring the local texture complexity.

[0116] The local contrast of the infrared image or SAR image is defined as the maximum value of its local standard deviation:

[0117] ;

[0118] ;

[0119] S4012. Normalize the local contrasts of the infrared image and the SAR image so that the sum of IRLocalContrast and SARLocalContrast is 1; obtain the weights of the local contrasts of the infrared image and the SAR image:

[0120] ;

[0121] In the formula, represents the weight of the infrared image based on the local contrast, represents the weight of the SAR image based on the local contrast.

[0122] S402. Calculate the information entropy of the infrared image and the SAR image, normalize the information entropy of the infrared image and the SAR image, and obtain the weights of the information entropy of the infrared image and the SAR image; specifically, it includes the following sub-steps:

[0123] S4021. Represent the number of occurrences of the gray value \(i\) in the infrared image or SAR image using a histogram; for the gray value of each pixel point in the gray image or , perform the following operations:

[0124] ;

[0125] where and represent the infrared histogram and the SAR histogram; \(i\) represents the gray value; represents the indicator function; when is equal to \(i\), is 1, otherwise 0; when is equal to \(i\), is 1, otherwise 0; represents the gray value of the pixel point with coordinates \((x,y)\) in the infrared image, represents the gray value of the pixel point with coordinates \((x,y)\) in the SAR image.

[0126] S4022. Perform normalization to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image:

[0127] ;

[0128] ;

[0129] where and are the probability distributions of each gray value in the infrared image and the SAR image respectively, and are the width and height of the infrared image respectively, and are the width and height of the SAR image respectively.

[0130] S4023. Use the Shannon information entropy formula, multiply the probability of each gray value by its logarithm to the base 2, and accumulate over all 256 possible gray values to calculate the infrared image information entropy \(H_{ir}\) and the SAR image information entropy \(H_{SAR}\);

[0131] ;

[0132] ;

[0133] The range of information entropy is from 0 to the maximum value of 8 bits, and this range reflects the amount of information in the image from completely disordered (the gray values of each pixel are random and have equal probabilities) to completely ordered (all pixels have the same gray value); when all pixels have the same gray value (i.e., the image is completely uniform), H ir = 0, H SAR = 0; when the probabilities of each gray value occurrence are equal, the information entropy H ir = 8, H SAR = 8, reaching the maximum value.

[0134] S4024. Normalize the information entropy of the infrared image and the SAR image so that H the sum of ir and H SAR is 1, obtaining the weights of the information entropy of the infrared image and the SAR image:

[0135] ;

[0136] In the formula, represents the weight of the infrared image based on information entropy, represents the weight of the SAR image based on information entropy.

[0137] S403. Convert the RGB format file of the infrared image to YUV, and make the brightness value of the Y channel of the infrared image Y ir be superimposed with the brightness value of the Y channel of the SAR image Y SAR pixel by pixel for fusion to generate the registered and fused image Y fused, and the expression is as follows:

[0138] ;

[0139] In the formula, Y fused represents the registered and fused image, Y ir represents the brightness value of the Y channel of the infrared image, Y SAR is the brightness value of the Y channel of the SAR image; represents the weight of the infrared image based on local contrast, represents the weight of the SAR image based on local contrast; represents the weight of the infrared image based on information entropy, represents the weight of the SAR image based on information entropy.

[0140] In summary, through a simplified ground calibration process, the present invention only needs to calibrate the field of view angles and imaging parameters of the infrared camera and SAR. After zooming or parameter changes, the registration algorithm is run on the airborne computer to extract heterologous image descriptors, match the descriptors of the infrared image and the SAR image, and store the projection relationship of each pixel coordinate of the infrared and SAR as matrix C. When the focal length or parameter configuration remains unchanged, matrix C only needs to be calculated once and stored in the airborne computer.

[0141] Since the infrared and SAR use a synchronous triggering method to ensure that the infrared and SAR acquire data at the same moment. After the data is acquired, a new frame of infrared image and SAR image arrive at the airborne computer simultaneously. By simply calling matrix C, real-time registration of the infrared and SAR can be completed, effectively guaranteeing the real-time performance of the registration. After each frame of image is registered through matrix C, the infrared video format is converted from RGB to YUV, and the corresponding brightness value of the SAR is superimposed on the Y channel. The local contrast and information entropy between the infrared and SAR are calculated, and the fusion ratio of the infrared Y and SAR brightness values is dynamically adjusted according to the local contrast and information entropy to maximize the information content of the fused image.

[0142] The method of the present invention is applicable to image registration under different resolutions and conditions where the infrared camera and SAR have variable focal lengths or variable parameter configurations, especially applicable to fields such as image registration between a variable focal length infrared camera and a SAR image of an integrated optoelectronic radar payload on an unmanned aerial vehicle, with broad application prospects and practical application value, significantly reducing the workload of registration calibration.

[0143] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in the disclosure of the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and no limitation is imposed herein.

[0144] The above specific embodiments do not constitute a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for registering and fusing infrared images and SAR images during dynamic flight of unmanned aerial vehicles, characterized in that: The specific steps include: S1. Ground calibration: measure and calibrate the relationship between the focal length of the infrared camera and the infrared Y-direction field of view angle on the ground, record the relationship between the field of view angle and the focal length as a file, and store it on the camera control board; S2. Rough matching of ground pixel resolution: The camera control board controls the movement of the infrared camera through the PID algorithm until the field of view angle of a single pixel in the infrared image matches the field of view angle of a single pixel in the SAR image; S3. Image descriptor extraction and registration: Extract heterogeneous image descriptors of infrared image and SAR image, match the descriptors, store the projection relationship of each pixel coordinate of infrared image and SAR image as matrix C; call matrix C to complete registration; heterogeneous image descriptors of infrared image and SAR image include phase consistency descriptor and gradient descriptor; match by weighted correlation distance method; the specific method is as follows: S301. Calculate the phase consistency descriptor of the heterogeneous image, which is used to measure the phase of a specific frequency component in the heterogeneous image; for each pixel, the phase can be calculated by the following formula: ； in, are pixel coordinates, w are different scales, is the phase at scale w, M represents the number of sizes; S302. Calculate the gradient descriptor of the heterogeneous image; The gradient direction descriptor of the heterogeneous image is as follows: ； ； in, and are the gradients in the x and y directions when the scale is m, respectively, and 𝐼 is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as: ； S303. Set the statistical scale to m, and take the (x, y) pixel as the center and the gradient histogram of the surrounding 10×10 pixels as follows: ； S304. Based on the image target size, take M / 2 scales upward and M / 2 scales downward, for a total of M scales; ; Phase consistency descriptor at the mth scale , gradient descriptor , gradient histogram It constitutes the descriptor vector at the mth scale : ； S305. Use the weighted correlation distance method to perform matching; the expression is as follows: ； Where WCD represents the weighted correlation distance, which is the weighted sum of the differences between the infrared image and the SAR image descriptor vectors from scale 1 to scale M; m is the scale of the descriptor; M represents the number of sizes; and Respectively represent the multi-scale descriptor vector of each feature point in the infrared image and the SAR image; is the weight of the mth scale, expressed as: ; is the descriptor vector of the mth scale and The variance of When the WCD value is greater than 0.5, it is considered and The match is successful; the projection relationship between the infrared image and each pixel coordinate in the SAR image is stored as a matrix C; S4. Image fusion: According to the registered image, the infrared image format is converted from RGB to YUV, and the corresponding brightness value of the SAR image is superimposed on the Y channel. The local contrast and information entropy of the infrared image and SAR image are calculated, and the fusion ratio of the infrared image and SAR image is dynamically adjusted to generate a registered fused image.

2. The method for registering and fusing infrared images and SAR images in dynamic flight of a UAV according to claim 1, characterized in that: The measurement and calibration method in step S1 specifically includes: using a reentrant infrared SAR dual-light common field light pipe to gradually measure the Y-direction field of view angle of the infrared camera; the reentrant infrared SAR dual-light common field of view light pipe contains a crosshair, and the step length is 0.1m.

3. The method for registering and fusing infrared images and SAR images in dynamic flight of a UAV according to claim 1, characterized in that: The method for adjusting the field of view angle of a single pixel of the infrared image to match the field of view angle of a single pixel of the SAR image in step S2 is specifically as follows: The onboard computer calculates the focal length of the infrared camera when each pixel of the infrared camera has the same resolution as the ground pixel of the SAR image based on the flight altitude of the UAV and the azimuth and pitch angle of the optoelectronic payload. ; The camera control board uses the PID algorithm to control the infrared camera to adjust to the focal length , until the field of view of a single infrared pixel matches the field of view of a single SAR pixel.

4. The method for registering and fusing infrared images and SAR images in dynamic flight of a UAV according to claim 1, characterized in that: The step S4 specifically includes the following sub-steps: S401. Calculate the local contrast of the infrared image and the SAR image; normalize the local contrast of the infrared image and the SAR image to obtain the weight of the local contrast of the infrared image and the SAR image; S402. Calculate the information entropy of the infrared image and the information entropy of the SAR image, normalize the information entropy of the infrared image and the SAR image, and obtain the weights of the information entropy of the infrared image and the SAR image; S403. Convert the RGB format file of the infrared image to YUV, and make the Y channel brightness value of the infrared image Y ir and Y channel brightness value of SAR image Y SAR Overlay and fuse pixel by pixel to generate a registered fused image Y fused, the expression is as follows: ； In the formula, Y fused means to register the fused image. Y ir represents the Y channel brightness value of the infrared image, Y SAR is the Y channel brightness value of the SAR image; represents the infrared image weight based on local contrast, represents the SAR image weight based on local contrast; represents the infrared image weight based on information entropy, Represents the SAR image weight based on information entropy.

5. The method for registering and fusing infrared images and SAR images in dynamic flight of a UAV according to claim 4, characterized in that: The step S401 specifically includes the following sub-steps: S4011. Determine a local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel value in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast; The calculation formulas for the local standard deviation of infrared images and SAR images are as follows: ； ； In the formula, is the infrared image at coordinates ( x , y ), is the SAR image at coordinates ( x , y ), is the local area Ω in the infrared image or SAR image f The average pixel value, |Ω f ∣ represents the local area Ω in the infrared image or SAR image f The total number of pixels; The local contrast expression of infrared image or SAR image is as follows: ； ； S4012. Normalize the local contrast of the infrared image and the SAR image so that the sum of IRLocalContrast and SARLocalContrast is 1; obtain the weight of the local contrast of the infrared image and the SAR image: ； In the formula, represents the infrared image weight based on local contrast, Represents the SAR image weight based on local contrast.

6. The method for registering and fusing infrared images and SAR images in dynamic flight of a UAV according to claim 5, characterized in that: The step S402 specifically includes the following sub-steps: S4021. Use a histogram to represent the number of times the gray value 𝑖 appears in the infrared image or SAR image; for each pixel in the gray image, the gray value or , do the following: ； In the formula, and Indicates infrared histogram and SAR histogram; i indicates gray value; represents the indicator function; when When it is equal to i, is 1, otherwise it is 0; When it is equal to i, is 1, otherwise it is 0; Represents the grayscale value of the pixel with coordinates (x, y) in the infrared image. Represents the grayscale value of the pixel with coordinates (x, y) in the SAR image; S4022. Perform normalization to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image: ； ； In the formula, and are the probability distribution of each gray value of infrared image and SAR image respectively, and are the width and height of the infrared image, and are the width and height of the SAR image respectively; S4023. Use the Shannon information entropy formula to multiply the probability of each gray value by its logarithm with base 2, and accumulate all 256 possible gray values ​​to calculate the infrared image information entropy 𝐻ir and the SAR image information entropy 𝐻SAR; the calculation formula is as follows: ； ； S4024. Normalize the information entropy of the infrared image and the SAR image so that H ir and H The sum of SAR is 1, and the weights of the information entropy of the infrared image and the SAR image are obtained: ； In the formula, represents the infrared image weight based on information entropy, Represents the SAR image weight based on information entropy.

7. The method for registering and fusing infrared images and SAR images in dynamic flight of a UAV according to claim 6, characterized in that: The value range of the information entropy is 0 to 8 bits; when all pixels have the same gray value, H ir=0, H SAR=0; when the probability of each gray value appearing is equal, the information entropy H ir=8, H SAR=8.

Images