Visible Light and Infrared Image Registration and Fusion Method for UAV Optoelectronic Turret

Abstract

The present invention relates to the technical field of image registration of unmanned aerial vehicle (UAV) payload systems, and particularly to a visible light and infrared image registration and fusion method for a UAV optoelectronic turret. It includes measuring and calibrating the relationship between the focal length of a visible light camera and the vertical field of view angle of the visible light on the ground, recording the relationship between the field of view angle and the focal length as a file, and storing it on the camera control board; the camera control board controls the movement of the visible light camera through a PID algorithm until the field of view angle of a single pixel of the visible light image matches the field of view angle of a single pixel of the infrared image; extracting and matching heterologous image descriptors, storing the projection relationship of each pixel coordinate as matrix A, and calling to complete the registration; calculating the local contrast and information entropy of the visible light image and the infrared image, dynamically adjusting the fusion ratio, and generating a registered and fused image. The advantages are as follows: only simple calibration is required on the ground; after changing the focal length, the registration algorithm runs on the on-board computer, and the combination of multi-scale descriptors improves the efficiency and accuracy of feature matching.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration for UAV payload systems, and particularly to a visible light and infrared image registration and fusion method for an optoelectronic turret of a UAV. Background Art

[0002] Image registration is an important technology in the fields of computer vision and image processing. It aims to align images acquired at different times, from different perspectives, or by different devices for comparison, analysis, or fusion. Traditional image registration methods usually require complex ground calibration processes, which include precise measurements of the offset of the optical axis centers of visible light and infrared radars, edge distortion, and pixel mapping relationships. These calibration processes are often time-consuming and laborious and are only applicable to the case of fixed-focus lenses. However, in the integrated payload system of an airborne optoelectronic radar on a UAV, the visible light is equipped with a zoom lens, and the ground resolution of the infrared radar is usually adjustable, which makes the traditional calibration methods no longer applicable because they need to be calibrated separately for each focal length, resulting in a huge workload.

[0003] In the prior art, the registration of visible light and infrared images usually requires the execution of cumbersome ground calibration procedures, including calibrating the offset of the optical axis centers of visible light and infrared, imaging mode differences, pixel mapping relationships, etc. The calibration process is complex and only applicable to the case where both visible light and infrared are fixed-focus lenses. The visible light camera of the integrated optoelectronic radar payload has a variable focal length, and the infrared system has variable parameter configurations. The workload of calibrating the center offset, edge distortion, pixel mapping, etc. for each focal length or parameter configuration is huge. The main limitations of the prior art are its complex ground calibration process and the lack of an image registration solution applicable to zoom lenses; the lack of an algorithm that can adapt to the focal length changes of the variable focal length visible light camera and the variable ground resolution infrared radar of the airborne optoelectronic radar on a UAV and achieve fast and accurate registration.

[0004] In addition, due to the dynamic nature of the UAV platform and the real-time requirements of tasks, traditional image registration methods face challenges in practical applications. When a UAV is performing a monitoring task, the change of the camera focal length is normal, which requires the image registration algorithm to be able to quickly adapt to the change of the visible light focal length and the ground resolution transformation of the infrared radar to achieve real-time processing. Therefore, there is an urgent need for an image registration method applicable to a variable focal length visible camera and a variable focal length infrared camera of an airborne optoelectronic turret on a UAV. Summary of the Invention

[0005] In order to solve the above problems, the present invention provides a visible light and infrared image registration and fusion method for an optoelectronic turret of a UAV.

[0006] The object of the present invention is to provide a visible light and infrared image registration and fusion method for an optoelectronic turret of a UAV, which specifically includes the following steps:

[0007] S1. Ground calibration: Measure and calibrate the relationship between the focal length of the visible light camera and the field of view angle in the Y direction of the visible light, and the relationship between the focal length of the infrared camera and the field of view angle in the Y direction of the infrared camera on the ground. Record the relationship between the field of view angle and the focal length as a file and store it on the camera control board;

[0008] S2. Coarse matching of field of view angles: The camera control board controls the movement of the visible light camera or the infrared camera through the PID algorithm so that the field of view angle of a single pixel in the visible light image is equal to the field of view angle of a single pixel in the infrared image;

[0009] S3. Extraction and registration of image descriptors: Extract the heterogeneous image descriptors of the visible light image and the infrared image, match the descriptors, and store the projection relationship of each pixel coordinate of the visible light image and the infrared image as matrix A; Call matrix A to complete the registration;

[0010] S4. Image fusion: According to the registered images, convert the format of the visible light image from RGB to YUV, and superimpose the corresponding brightness values of the infrared image on the Y channel. Calculate the local contrast and information entropy of the visible light image and the infrared image, and dynamically adjust the fusion ratio of the visible light image and the infrared image to generate a registered and fused image.

[0011] Preferably, the measurement and calibration method in step S1 specifically includes: using a folding visible-infrared dual-light co-field optical tube to gradually measure the field of view angle in the Y direction of the visible light camera and the field of view angle in the Y direction of the infrared camera; The folding visible-infrared dual-light co-field optical tube contains a crosshair, and the stepping length is 0.1 m.

[0012] Preferably, the method for adjusting the field of view angle of a single pixel in the visible light image to be equal to the field of view angle of a single pixel in the infrared image in step S2 is as follows:

[0013] When the UAV is performing a task, if the visible lens is the main one and the infrared is fused, the following steps are executed: Taking the visible lens as the main lens, when the ground station sends the large field of view and small field of view commands, only the visible lens responds; After the ground station control ends, the visible lens is in a stationary state. At this time, read the focal length value g of the visible light CCD ; According to the focal length value g of the visible light CCD 、the visible pixel size and the infrared pixel size, calculate the infrared focal length under the condition of the same field of view for a single pixel. The camera control board controls the repeated movement of the infrared camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length;

[0014] When the UAV is performing a task, if the infrared lens is the main one and the visible lens is fused, the following steps are executed: Taking the infrared lens as the main lens, when the ground station sends the large field of view and small field of view commands, only the infrared lens responds; After the ground station control ends, the infrared lens is in a stationary state. At this time, read the focal length value g of the infrared lens IR; According to the focal length value g of the infrared lens IR , based on the visible pixel size and the infrared pixel size, calculate the visible light focal length under the condition of the same field of view for a single pixel. The camera control board controls the repeated movement of the visible light camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length.

[0015] Preferably, the heterologous image descriptors of the visible light image and the infrared image in step S3 include phase consistency descriptors and gradient descriptors; the method of weighted correlation distance is used for matching; the specific method is as follows:

[0016] S301. Calculate the phase consistency descriptor of the heterologous image to measure the phase of specific frequency components in the heterologous image; for each pixel, the phase can be calculated by the following formula:

[0017] ;

[0018] where, is the pixel coordinate, w is different scales, is the phase at scale w, and M represents the number of sizes;

[0019] S302. Calculate the gradient descriptor of the heterologous image;

[0020] The gradient direction descriptor of the heterologous image is as follows:

[0021] ; ;

[0022] where, and are the gradients in the x - direction and y - direction at scale m respectively, I is the brightness of the image; thus, the gradient descriptor of the heterologous image is expressed as:

[0023]

[0024] ;

[0025] In the formula, represents the direction of the gradient, that is, the direction of the brightness change of the image at the point (x,y);

[0026] S303. Set the statistical scale as m, and the gradient histogram of 10×10 pixels around the pixel (x,y) is as follows:

[0027] ;

[0028] S304. Based on the image target size, take M / 2 scales upward and M / 2 scales downward, for a total of M scales; ; The phase consistency descriptor at the m-th scale , the gradient descriptor , the histogram of gradients constitute the descriptor vector at the m-th scale :

[0029] ;

[0030] S305. Matching is performed by the method of weighted correlation distance; the expression is as follows:

[0031] ;

[0032] In the formula, WCD represents the weighted correlation distance, that is, the weighted sum of the differences between the visible light image and the infrared image descriptor vectors from scale 1 to scale M; m is the scale of the descriptor; M represents the number of scales; and respectively represent the multi-scale descriptor vectors at the m-th scale in the visible light image and the infrared image; is the weight of the m-th scale, expressed as: ; is the descriptor vector of the m-th scale and are the variances of;

[0033] When the WCD value is greater than 0.5, it is considered that and are successfully matched; the projection relationship of each pixel coordinate in the visible light image and the infrared image is stored as matrix A.

[0034] Preferably, step S4 specifically includes the following sub-steps:

[0035] S401. Calculate the local contrast of the visible light image and the infrared image; normalize the local contrast of the visible light image and the infrared image to obtain the weights of the local contrast of the visible light image and the infrared image;

[0036] S402. Calculate the information entropy of the visible light image and the infrared image, normalize the information entropy of the visible light image and the infrared image to obtain the weights of the information entropy of the visible light image and the infrared image;

[0037] S403. Convert the RGB format file of the visible light image to YUV, and superpose the Y-channel brightness value Y ccd of the visible light image with the Y-channel brightness value Y ir of the infrared image, and perform pixel-by-pixel fusion to generate the registered and fused image Y fused, the expression is as follows:

[0038] ;

[0039] In the formula, Y fused represents the registered and fused image, Y ccd represents the Y-channel brightness value of the visible light image, Y ir is the Y-channel brightness value of the infrared image; represents the weight of the visible light image based on local contrast, represents the weight of the infrared image based on local contrast; represents the weight of the visible light image based on information entropy, represents the weight of the infrared image based on information entropy.

[0040] Preferably, step S401 specifically includes the following sub-steps:

[0041] S4011. Determine the local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel values in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast;

[0042] The local standard deviation calculation formulas for the visible light image and the infrared image are as follows:

[0043] ;

[0044] ;

[0045] In the formula, is the pixel value of the visible image at the coordinate ( x , y ), is the pixel value of the infrared image at the coordinate ( x , y ), is the average pixel value of the local area Ω f in the visible light image or the infrared image, |Ω f | represents the total number of pixels in the local area Ω f in the visible light image or the infrared image;

[0046] The local contrast expressions for the visible light image or the infrared image are as follows:

[0047] CCDLocalContrast = max( );

[0048] IRLocalContrast = max( );

[0049] S4012. Normalize the local contrast of the visible light image and the infrared image so that the sum of CCDLocalContrast and IRLocalContrast is 1; obtain the weights of the local contrast of the visible light image and the infrared image:

[0050] ;

[0051] In the formula, represents the weight of the visible light image based on the local contrast, represents the weight of the infrared image based on the local contrast.

[0052] Preferably, step S402 specifically includes the following sub-steps:

[0053] S4021. Use a histogram to represent the number of times the gray value i appears in the visible light image or the infrared image; for the gray value of each pixel point in the gray image or , perform the following operations:

[0054] ;

[0055] In the formula, and represent the visible histogram and the infrared histogram; i represents the gray value; represents the indicator function; when ; when ; represents the gray value of the pixel point with coordinates (x, y) in the visible image, represents the gray value of the pixel point with coordinates (x, y) in the infrared image;

[0056] S4022. Perform normalization processing to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image:

[0057] ;

[0058] ;

[0059] In the formula, and are the probability distributions of each gray value of the visible light image and the infrared image respectively, and are the width and height of the visible light image respectively, and are the width and height of the infrared image respectively;

[0060] S4023. Use the Shannon information entropy formula to multiply the probability of each grayscale value by its base-2 logarithm and sum over all 256 possible grayscale values to calculate the visible light image information entropy H Information entropy of CCD and infrared images H ir; The calculation formula is as follows:

[0061] ;

[0062] ;

[0063] S4024. Normalize the information entropy of the visible light image and the infrared image so that H the sum of CCD and H ir is 1 to obtain the weights of the visible light image and the infrared image information entropy:

[0064] ;

[0065] In the formula, represents the weight of the visible light image based on information entropy, represents the weight of the infrared image based on information entropy.

[0066] Preferably, the value range of the information entropy is 0 to 8 bits; when all pixels have the same grayscale value, H CCD = 0, H ir = 0; when the probability of each grayscale value appearing is equal, the information entropy H CCD = 8, H ir = 8.

[0067] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0068] The present invention only needs to simply calibrate the field of view angles of visible and infrared on the ground. After zooming, on the airborne computer, run the registration algorithm, combine multi-scale descriptors and an optimized search strategy to improve the efficiency and accuracy of feature matching. It is applicable to the registration of visible and infrared images under different resolutions and conditions where the visible and infrared radars are zoom lenses, especially applicable to the image registration in fields such as the integrated payload of an unmanned aerial vehicle's airborne optoelectronic radar with a variable focal length visible camera and a variable ground resolution infrared radar, etc., and has broad application prospects and practical application value, significantly reducing the workload of registration calibration. Description of the Drawings

[0069] Figure 1 is a flowchart of a visible and infrared image registration and fusion method for an unmanned aerial vehicle optoelectronic turret according to an embodiment of the present invention.

[0070] Figure 2It is a visible light image taken by a drone aerial photography according to an embodiment of the present invention.

[0071] Figure 3 It is an infrared image taken by a drone aerial photography according to an embodiment of the present invention.

[0072] Figure 4 It is a fused registration image generated after the fusion of the visible light image and the infrared image on the drone according to an embodiment of the present invention. Detailed implementation manners

[0073] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are denoted by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, their detailed descriptions will not be repeated.

[0074] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation to the present invention.

[0075] The present invention aims to solve the limitations of the prior art and provides a new image registration method, which is applicable to the integrated payload system of an airborne optoelectronic radar of a drone, can reduce the calibration workload, and improve the efficiency and accuracy of image registration. By performing simple field of view calibration on the ground and combining multi-scale descriptors and an optimized search strategy, the registration algorithm can be run in real time on an airborne computer to meet the requirements of real-time image processing of the drone.

[0076] See Figure 1 , the present invention provides a method for registering and fusing visible light and infrared images for an optoelectronic turret of a drone, specifically including the following steps:

[0077] S1. Ground calibration: Install a visible light camera and an infrared camera on the airborne optoelectronic turret of the drone; Measure and calibrate the relationship between the focal length of the visible light camera and the field of view angle in the Y direction of the visible light, and the relationship between the focal length of the infrared camera and the field of view angle in the Y direction of the infrared; Record the relationship between the field of view angle and the focal length as a file and store it on the camera control board;

[0078] The measurement and calibration method specifically includes: Using a folding visible-infrared dual-light co-viewing light pipe (with a crosshair inside the light pipe and a step length of 0.1 m) to gradually measure the field of view angle in the Y direction of the visible light camera and the field of view angle in the Y direction of the infrared camera;

[0079] Specifically, the optoelectronic turret is internally equipped with measured azimuth and elevation encoders. Assume that the focal length of the visible light camera lens is CCD_F at this time, the elevation angle of the optoelectronic turret is locked at 0, align the upper edge of the visible light camera with the crosshair of the light pipe, and read the azimuth angle CCDup of the optoelectronic turret at this time. Rotate the optoelectronic turret so that the lower edge of the visible light camera aligns with the crosshair of the light pipe, and read the azimuth angle CCDdown of the optoelectronic turret at this time. Then, when the focal length of the visible light camera is CCD_F, the field of view angle in the Y direction of the visible light is CCD_Y = CCDup - CCDdown; record the corresponding relationship between the focal length of the visible light lens and the field of view angle into the array c. Similarly, measure the field of view angle in the Y direction of the infrared camera, IR_Y = IRup - IRdown; record the corresponding relationship between the focal length of the infrared lens and the field of view angle into the array d, and store it in the optoelectronic turret recorder on the UAV.

[0080] S2. Coarse matching of the field of view angle: The camera control board controls the movement of the visible light camera or the infrared camera through the PID algorithm to make the field of view angle of a single pixel in the visible light image equal to the field of view angle of a single pixel in the infrared image. The specific operation is as follows:

[0081] If the field of view angles of each pixel in the visible light image and the infrared image are to be the same, the derivation of the focal length relationship between the visible light lens and the infrared lens is as follows:

[0082] Visible light pixel size: ; Infrared pixel size: ; The focal length of the visible light lens is fvi ; The focal length of the infrared lens is fir ;

[0083] The angular resolution θ can be expressed by the following formula: θ = pixel size / focal length;

[0084] The angular resolution of the visible light image is θ vis = dvis / fvis ; The angular resolution of the infrared image is ;

[0085] To make θ vis equal to θ ir , the following equation can be established: ; ; That is, in the case of a visible light pixel size of 2.5 μm and an infrared pixel size of 15 μm, that is, when the visible light focal length is 1 / 6 of the infrared focal length, the field of view angle of a single pixel in the visible light image is equal to the field of view angle of a single pixel in the infrared image.

[0086] The visible light camera and the infrared camera adopt an external trigger method to ensure that the visible light and the infrared are exposed at the same time.

[0087] Adjust the field of view angle of a single pixel in the visible light image to be equal to that of a single pixel in the infrared image, that is, the method of adjusting the visible focal length to be 1 / 6 of the infrared focal length is as follows:

[0088] When the UAV is performing a mission, if the visible lens is the main one and the infrared is fused, the following steps are executed: Taking the visible lens as the main lens, when the ground station sends the large field of view and small field of view commands, only the visible lens responds; after the ground station control ends, the visible lens is in a static state. At this time, read out the focal length value g of the visible light CCD ; According to the focal length value g of the visible light CCD , the visible pixel size and the infrared pixel size, calculate the infrared focal length under the condition of the same field of view for a single pixel. The camera control board controls the repeated movement of the infrared camera through the PID algorithm until the visible focal length is 1 / 6 of the infrared focal length;

[0089] When the UAV is performing a mission, if the infrared lens is the main one and the visible lens is fused, the following steps are executed: Taking the infrared lens as the main lens, when the ground station sends the large field of view and small field of view commands, only the infrared lens responds; after the ground station control ends, the infrared lens is in a static state. At this time, read out the focal length value g of the infrared lens IR ; According to the focal length value g of the infrared lens IR , according to the visible pixel size and the infrared pixel size, calculate the visible focal length under the condition of the same field of view for a single pixel. The camera control board controls the repeated movement of the visible camera through the PID algorithm until the visible focal length is 1 / 6 of the infrared focal length.

[0090] S3. Image descriptor extraction and registration: Extract the heterogeneous image descriptors of the visible light image and the infrared image, match the descriptors, and store the projection relationship of each pixel coordinate of the visible light image and the infrared image as matrix A; Call matrix A to complete the registration;

[0091] The heterogeneous image descriptors of the visible light image and the infrared image include phase consistency descriptors and gradient descriptors; The method of matching the descriptors is to combine the phase consistency descriptors and the gradient descriptors, extract the common feature descriptors of the visible light image and the infrared image, and use the method of weighted correlation distance (WCD) for matching; The specific method is as follows:

[0092] S301. Calculate the phase consistency descriptor of the heterogeneous image, which is used to measure the phase of specific frequency components in the heterogeneous image; For each pixel, the phase can be calculated by the following formula:

[0093] ;

[0094] Among them, is the pixel coordinate, w is different scales, is the phase at scale w, and M represents the number of scales (usually taking the value of 10);

[0095] S302. Calculate the gradient descriptor of the heterologous image;

[0096] The gradient direction descriptor of the heterologous image is as follows:

[0097] ; ;

[0098] Among them, and are the gradients in the x - direction and y - direction at scale m respectively, I is the brightness of the image; thus, the gradient descriptor of the heterologous image is expressed as:

[0099]

[0100] ;

[0101] In the formula, represents the direction of the gradient, that is, the direction of the brightness change of the image at the point (x, y);

[0102] S303. Set the statistical scale as m, and the gradient histogram of 10×10 pixels centered on the (x, y) pixel is as follows:

[0103] ;

[0104] S304. Based on the target size of the image, take M / 2 scales upward and M / 2 scales downward, a total of M scales. The phase consistency descriptor at the m - th scale , the gradient descriptor (gradient amplitude) , and the gradient histogram constitute the descriptor vector at the m - th scale:

[0105] ;

[0106] Both the visible - light image and the infrared image are operated according to the above - mentioned method, and a rich multi - scale descriptor vector and are respectively generated for each feature point in the visible - light image and the infrared image. This descriptor effectively describes the local structural information of the visible image and the infrared image and provides a solid foundation for feature matching.

[0107] S305. Perform matching using the method of Weighted Correlation Distance (WCD); the expression is as follows:

[0108] ;

[0109] In the formula, WCD represents the weighted correlation distance, that is, the weighted sum of the differences between the visible light image and the infrared image descriptor vectors from scale 1 to scale M; m is the scale of the descriptor (the value range of m is from 1 to M); M represents the number of sizes (usually taken as 10); and respectively represent the multi-scale descriptor vectors at the m-th scale in the visible light image and the infrared image; is the weight of the m-th scale, expressed as: ; is the descriptor vector of the m-th scale and is the variance of;

[0110] When the WCD value is greater than 0.5, it is considered that and are successfully matched; the projection relationship of each pixel coordinate in the visible light image and the infrared image is stored as matrix A.

[0111] Principle brief: After step S2, it is theoretically ensured that the visible and infrared fields of view are basically the same in the Y direction. However, due to the field of view angle calibration error in step S1 and the focal length control error in step S2, there are still slight differences in the visible and infrared fields of view, so registration operations are required. The calculation of the common feature descriptors of the visible light image and the infrared image is a key step in the matching of the visible light image and the infrared image, which provides a unique vector representation for each feature point of the visible light image and the infrared image.

[0112] Using the weighted correlation distance (WCD) method not only considers the Euclidean distance between feature vectors but also considers the correlation between feature dimensions. This method is particularly suitable for situations where there are internal connections between feature dimensions.

[0113] Since the visible and infrared use an external trigger method to ensure that the visible and infrared are exposed at the same time. After exposure, a new frame of visible light image and infrared image arrive at the airborne computer. Just by calling matrix A, the real-time registration of the visible and infrared can be completed, effectively guaranteeing the real-time performance of the registration.

[0114] S4. Image fusion: According to the registered images, convert the visible light image format from RGB to YUV, and overlay the corresponding brightness values of the infrared image on the Y channel. Calculate the local contrast and information entropy of the visible light image and the infrared image, and dynamically adjust the fusion ratio of the visible light image and the infrared image to generate a registered and fused image; specifically including the following sub-steps:

[0115] S401. Calculate the local contrast of the visible light image and the infrared image; normalize the local contrast of the visible light image and the infrared image to obtain the weights of the local contrast of the visible light image and the infrared image; specifically including the following sub-steps:

[0116] S4011. Determine the local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel values in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast.

[0117] Local standard deviation is a measure of the pixel value fluctuation in the local area The local standard deviation calculation formulas for the visible light image and the infrared image are as follows:

[0118] ;

[0119] ;

[0120] In the formula, is the pixel value of the visible image at coordinates ( x , y ), is the pixel value of the infrared image at coordinates ( x , y ), is the average pixel value of the local area Ω f in the visible light image or the infrared image, and ∣Ω f ∣ represents the total number of pixels in the local area Ω f in the visible light image or the infrared image.

[0121] The local standard deviation is calculated by summing the squares of the deviations between the pixel values in the local area Ω f and its average value μf and then taking the square root. This value reflects the degree of dispersion of pixel intensities in the feature area and is an indicator for measuring local texture complexity.

[0122] The local contrast of the visible light image or the infrared image is defined as the maximum value of its local standard deviation:

[0123] CCDLocalContrast = max( );

[0124] IRLocalContrast = max( );

[0125] S4012. Normalize the local contrast of the visible light image and the infrared image so that the sum of CCDLocalContrast and IRLocalContrast is 1; obtain the weights of the local contrast of the visible light image and the infrared image:

[0126] ;

[0127] wherein, represents the weight of the visible light image based on the local contrast, represents the weight of the infrared image based on the local contrast.

[0128] S402. Calculate the information entropy of the visible light image and the infrared image, normalize the information entropy of the visible light image and the infrared image, and obtain the weights of the information entropy of the visible light image and the infrared image; specifically, it includes the following sub-steps:

[0129] S4021. Use a histogram to represent the number of times the gray value i appears in the visible light image or the infrared image; for the gray value of each pixel point in the gray image or , perform the following operations:

[0130] ;

[0131] wherein, and represent the visible histogram and the infrared histogram; i represents the gray value; represents the indicator function; when ; when ; represents the gray value of the pixel point with coordinates (x, y) in the visible image, represents the gray value of the pixel point with coordinates (x, y) in the infrared image;

[0132] S4022. Perform normalization processing, convert the histogram into a probability distribution, and achieve this by dividing the frequency of each gray value by the total number of pixels in the image:

[0133] ;

[0134] ;

[0135] wherein, and are respectively the probability distributions of each gray value of the visible light image and the infrared image, and are the width and height of the visible light image respectively, and are the width and height of the infrared image respectively.

[0136] S4023. Use the Shannon information entropy formula to multiply the probability of each gray value by its logarithm to the base 2, and sum over all 256 possible gray values to calculate the information entropy of the visible light image H ccd and the information entropy of the infrared image H ir;

[0137] ;

[0138] ;

[0139] The range of the information entropy is from 0 to the maximum value of 8 bits, and this range reflects the amount of information of the image from completely disordered (the gray values of each pixel point are random and have equal probabilities) to completely ordered (all pixel points have the same gray value); when all pixels have the same gray value (i.e., the image is completely uniform), H ccd = 0, H ir = 0; when the probabilities of each gray value occurrence are equal, the information entropy H ccd = 8, H ir = 8, reaching the maximum value.

[0140] S4024. Normalize the information entropy of the visible light image and the infrared image so that H ccd and H ir sum to 1 to obtain the weights of the information entropy of the visible light image and the infrared image:

[0141] ;

[0142] In the formula, represents the weight of the visible light image based on the information entropy, represents the weight of the infrared image based on the information entropy.

[0143] S403. Convert the RGB format file of the visible light image to YUV, and superimpose the Y-channel brightness value Y ccd of the visible light image with the Y-channel brightness value Y ir of the infrared image, and perform pixel-by-pixel fusion to generate the registered and fused image Y fused, and the expression is as follows:

[0144] ;

[0145] In the formula, Y"fused" represents the registered and fused image, Y "ccd" represents the Y-channel brightness value of the visible light image, Y "ir" is the Y-channel brightness value of the infrared image; represents the weight of the visible light image based on local contrast, represents the weight of the infrared image based on local contrast; represents the weight of the visible light image based on information entropy, represents the weight of the infrared image based on information entropy.

[0146] In summary, through a simplified ground calibration process, the present invention only needs to calibrate the field of view angles and imaging parameters of the visible light camera and the infrared camera. After zooming or parameter changes, the registration algorithm is run on the airborne computer to extract heterogeneous image descriptors, match the descriptors of the visible light and infrared images, and store the projection relationship of each pixel coordinate of the visible light and infrared as matrix A. When the focal length or parameter configuration remains unchanged, matrix A only needs to be calculated once and stored in the airborne computer.

[0147] Since the visible light and the infrared use a synchronous triggering method to ensure that the visible light and the infrared acquire data at the same moment. After the data is acquired, a new frame of visible light image and infrared image arrive at the airborne computer simultaneously. Only by calling matrix A can the real-time registration of the visible light and the infrared be completed, effectively guaranteeing the real-time performance of the registration. After each frame of image is registered through matrix A, the visible light video format is converted from RGB to YUV, and the corresponding brightness value of the infrared is superimposed on the Y channel. Calculate the local contrast and information entropy of the visible light and the infrared, and dynamically adjust the fusion ratio of the visible light Y and the infrared brightness value according to the local contrast and information entropy to maximize the information content of the fused image.

[0148] The method of the present invention is applicable to image registration under different resolutions and conditions where the visible light camera and the infrared camera have variable focal lengths or variable parameter configurations, especially applicable to fields such as image registration between a variable focal length visible light camera and an infrared image of an integrated payload of an unmanned aerial vehicle airborne optoelectronic radar, etc., with broad application prospects and practical application value, significantly reducing the workload of registration calibration.

[0149] It should be understood that various forms of the processes shown above can be used, steps can be reordered, added or deleted. For example, the steps recorded in the disclosure of the present invention can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and no limitations are imposed herein.

[0150] The above specific embodiments do not constitute a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A visible light and infrared image registration and fusion method for an unmanned aerial vehicle optoelectronic turret, characterized in that: The specific steps include: S1. Ground calibration: measure and calibrate the relationship between the focal length of the visible light camera and the field angle in the Y direction of the visible light, and the relationship between the focal length of the infrared camera and the field angle in the Y direction of the infrared light, and record the relationship between the field angle and the focal length as a file and store it on the camera control board; S2. Coarse matching of field of view angle: The camera control board controls the movement of the visible light camera or infrared camera through the PID algorithm, so that the field of view angle of a single pixel of the visible light image is equal to the field of view angle of a single pixel of the infrared image; S3. Image descriptor extraction and registration: Extract heterogeneous image descriptors of visible light image and infrared image, match the descriptors, store the projection relationship of each pixel coordinate of visible light image and infrared image as matrix A; call matrix A to complete the registration; heterogeneous image descriptors of visible light image and infrared image include phase consistency descriptor and gradient descriptor; match by weighted correlation distance method; the specific method is as follows: S301. Calculate the phase consistency descriptor of the heterogeneous image, which is used to measure the phase of a specific frequency component in the heterogeneous image; for each pixel, the phase can be calculated by the following formula: ； in, are pixel coordinates, w are different scales, is the phase at scale w, M represents the number of sizes; S302. Calculate the gradient descriptor of the heterogeneous image; The gradient direction descriptor of the heterogeneous image is as follows: ； ； in, and They are the gradients in the x and y directions when the scale is m, I is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as: ； In the formula, Indicates the direction of the gradient, that is, the direction of the brightness change of the image at the point (x, y); S303. Set the statistical scale to m, and take the (x, y) pixel as the center and the gradient histogram of the surrounding 10×10 pixels as follows: ； S304. Based on the image target size, take M / 2 scales upward and M / 2 scales downward, for a total of M scales; ; Phase consistency descriptor at the mth scale , gradient descriptor , gradient histogram It constitutes the descriptor vector at the mth scale : ； S305. Use the weighted correlation distance method to perform matching; the expression is as follows: ； Where WCD represents the weighted correlation distance, which is the weighted sum of the differences between the descriptor vectors of the visible light image and the infrared image from scale 1 to scale M; m is the scale of the descriptor; M represents the number of scales; and Represent the multi-scale descriptor vector at the mth scale in the visible light image and infrared image respectively; is the weight of the mth scale, expressed as: ; is the descriptor vector of the mth scale and The variance of When the WCD value is greater than 0.5, it is considered and The match is successful; the projection relationship between the coordinates of each pixel in the visible light image and the infrared image is stored as a matrix A; S4. Image fusion: According to the registered image, the visible light image format is converted from RGB to YUV, and the corresponding brightness value of the infrared image is superimposed on the Y channel. The local contrast and information entropy of the visible light image and the infrared image are calculated, and the fusion ratio of the visible light image and the infrared image is dynamically adjusted to generate a registered fused image.

2. The visible light and infrared image registration and fusion method for an unmanned aerial vehicle optoelectronic turret according to claim 1 is characterized in that: The measurement and calibration method in step S1 specifically includes: using a reentrant visible infrared dual light common field light tube to gradually measure the Y direction field angle of the visible light camera and the Y direction field angle of the infrared camera; the reentrant visible infrared dual light common field light tube contains a crosshair, and the step length is 0.1m.

3. The visible light and infrared image registration and fusion method for the optoelectronic turret of a UAV according to claim 2 is characterized by: The method for adjusting the field of view angle of a single pixel of the visible light image to be equal to the field of view angle of a single pixel of the infrared image in step S2 is specifically as follows: When the UAV performs a mission, if the visible lens is used as the main lens and infrared is integrated, the following steps are performed: when the visible lens is used as the main lens and the ground station sends a large field of view or small field of view command, only the visible lens responds; after the ground station control ends, the visible lens is in a static state. At this time, the focal length value g of the visible light is read out. CCD ; According to the focal length of visible light g CCD , visible pixel size and infrared pixel size, calculate the infrared focal length of a single pixel under the same field of view, and the camera control board controls the repeated movement of the infrared camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length; When the UAV is performing a mission, if the infrared lens is used as the main lens and the visible lens is integrated, the following steps are performed: when the infrared lens is used as the main lens and the ground station sends a large field of view or a small field of view command, only the infrared lens responds; after the ground station control ends, the infrared lens is in a static state. At this time, the focal length value g of the infrared lens is read out. IR ; According to the focal length value g of the infrared lens IR , according to the visible pixel size and the infrared pixel size, the visible light focal length of a single pixel under the same field of view is calculated. The camera control board controls the repeated movement of the visible light camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length.

4. The visible light and infrared image registration and fusion method for an unmanned aerial vehicle optoelectronic turret according to claim 1 is characterized in that: The step S4 specifically includes the following sub-steps: S401. Calculate the local contrast of the visible light image and the infrared image; normalize the local contrast of the visible light image and the infrared image to obtain the weight of the local contrast of the visible light image and the infrared image; S402. Calculate the information entropy of the visible light image and the information entropy of the infrared image, normalize the information entropy of the visible light image and the infrared image, and obtain the weights of the information entropy of the visible light image and the infrared image; S403. Convert the RGB format file of the visible light image to YUV, and make the Y channel brightness value of the visible light image Y CCD and infrared image Y channel brightness value Y IR superposition, pixel-by-pixel fusion, to generate a registered fused image Y fused, the expression is as follows: ； In the formula, Y fused means to register the fused image. Y CCD represents the Y channel brightness value of the visible light image. Y ir is the Y channel brightness value of the infrared image; represents the visible light image weight based on local contrast, represents the infrared image weight based on local contrast; represents the weight of the visible light image based on information entropy, Represents the infrared image weight based on information entropy.

5. The visible light and infrared image registration and fusion method for the optoelectronic turret of a UAV according to claim 4 is characterized in that: The step S401 specifically includes the following sub-steps: S4011. Determine a local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel value in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast; The calculation formulas for the local standard deviation of visible light images and infrared images are as follows: ； ； In the formula, is the visible image at coordinates ( x , y ), is the infrared image at coordinates ( x , y ), is the local area Ω in the visible light image or infrared image f The average pixel value, |Ω f ∣ represents the local area Ω in the visible light image or infrared image f The total number of pixels; The local contrast expression of a visible light image or infrared image is as follows: CCDLocalContrast=max( ); IRLocalContrast=max( ); S4012. Normalize the local contrast of the visible light image and the infrared image so that the sum of CCDLocalContrast and IRLocalContrast is 1; obtain the weight of the local contrast of the visible light image and the infrared image: ； In the formula, represents the visible light image weight based on local contrast, Represents the infrared image weight based on local contrast.

6. The visible light and infrared image registration and fusion method for the optoelectronic turret of a UAV according to claim 5 is characterized by: The step S402 specifically includes the following sub-steps: S4021. Use a histogram to represent the number of times the gray value i appears in the visible light image or infrared image; for each pixel in the gray image, the gray value or , do the following: ； In the formula, and Represents visible histogram and infrared histogram; i represents grayscale value; represents the indicator function; when ;when ; Represents the grayscale value of the pixel with coordinates (x, y) in the visible image. Represents the grayscale value of the pixel with coordinates (x, y) in the infrared image; S4022. Perform normalization to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image: ； ； In the formula, and are the probability distribution of each gray value of the visible light image and infrared image, and are the width and height of the visible light image, respectively. and are the width and height of the infrared image respectively; S4023. Using the Shannon information entropy formula, multiply the probability of each gray value by its logarithm with base 2, and accumulate all 256 possible gray values ​​to calculate the information entropy of the visible light image. H CCD and infrared image information entropy H ir; the calculation formula is as follows: ； ； S4024. Normalize the information entropy of the visible light image and the infrared image so that H CCD and H ir is 1, and the weights of the information entropy of the visible light image and the infrared image are obtained: ； In the formula, represents the weight of the visible light image based on information entropy, Represents the infrared image weight based on information entropy.

7. The visible light and infrared image registration and fusion method for the optoelectronic turret of a UAV according to claim 6 is characterized by: The value range of the information entropy is 0 to 8 bits; when all pixels have the same grayscale value, H CCD = 0, H ir = 0; when the probability of each gray value appearing is equal, the information entropy H CCD = 8, H ir =8.

Images