Method for registering and fusing visible light and SAR images during dynamic flight of unmanned aerial vehicle

Abstract

The present invention relates to the technical field of image registration of UAV payload systems, and particularly to a method for registering and fusing visible light and SAR images during the dynamic flight of a UAV. It includes measuring and calibrating the relationship between the focal length of the visible light camera and the vertical field of view angle of the visible light on the ground, recording the relationship between the field of view angle and the focal length as a file, and storing it on the camera control board; the camera control board controls the movement of the visible light camera through the PID algorithm until the field of view angle of a single pixel of the visible light image matches the field of view angle of a single pixel of the SAR image; extracting and matching the heterologous image descriptors, storing the projection relationship of each pixel coordinate as matrix B, and calling to complete the registration; calculating the local contrast and information entropy of the visible light image and the SAR image, dynamically adjusting the fusion ratio, and generating the registered and fused image. The advantages are as follows: only simple calibration is required on the ground; after changing the focal length, the registration algorithm runs on the on-board computer, and the multi-scale descriptors are combined to improve the efficiency and accuracy of feature matching.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration of UAV payload systems, and particularly to a method for registering and fusing visible light and SAR images during the dynamic flight of a UAV. Background Art

[0002] Traditional image registration methods usually require a complex ground calibration process, which includes precise measurement of the offset of the optical axis centers of visible light and SAR radars, edge distortion, and pixel mapping relationships. These calibration processes are often time-consuming and laborious, and are only applicable to the case of fixed-focus lenses. However, in the integrated optoelectronic radar payload system of a UAV, the visible light is equipped with a zoom lens, and the ground resolution of the SAR radar can usually be adjusted, which makes the traditional calibration methods no longer applicable because they need to be calibrated separately for each focal length, resulting in a huge workload.

[0003] In addition, due to the dynamic nature of the UAV platform and the real-time requirements of the mission, traditional image registration methods face challenges in practical applications. When a UAV is performing a monitoring mission, it is normal for the camera focal length to change, which requires the image registration algorithm to be able to quickly adapt to the change in the visible light focal length and the change in the ground resolution of the SAR radar to achieve real-time processing.

[0004] In the prior art, the registration of visible light and SAR images usually requires the execution of a cumbersome ground calibration procedure, including calibrating the offset of the optical axis centers of visible light and SAR, imaging mode differences, pixel mapping relationships, etc. The calibration process is complex and only applicable to the case where the visible light is a fixed-focus lens and the SAR is a fixed parameter configuration. The visible light camera of the integrated optoelectronic radar payload is a zoom lens, and the SAR system is a variable parameter configuration. The workload of calibrating the center offset, imaging mode adaptation, pixel mapping, etc. for each focal length or parameter configuration is huge. The main limitations of the prior art are its complex ground calibration process and the lack of an image registration solution applicable to zoom lenses; the lack of an algorithm that can adapt to the focal length changes of the variable-focus visible camera and the variable ground resolution SAR radar of the UAV-borne integrated optoelectronic radar payload and achieve fast and accurate registration. Summary of the Invention

[0005] In order to solve the above problems, the present invention provides a method for registering and fusing visible light and SAR images during the dynamic flight of a UAV.

[0006] The object of the present invention is to provide a method for registering and fusing visible light and SAR images during the dynamic flight of a UAV, which specifically includes the following steps:

[0007] S1. Ground calibration: Measure and calibrate the relationship between the focal length of the visible light camera and the field of view angle in the Y direction of the visible light on the ground, record the relationship between the field of view angle and the focal length as a file, and store it on the camera control board;

[0008] S2. Coarse matching of ground pixel resolution: The camera control board controls the movement of the visible light camera through the PID algorithm until the field of view angle of a single pixel in the visible light image matches the field of view angle of a single pixel in the SAR image;

[0009] S3. Extraction and registration of image descriptors: Extract the heterogeneous image descriptors of the visible light image and the SAR image, match the descriptors, and store the projection relationship of each pixel coordinate of the visible light image and the SAR image as matrix B; Call matrix B to complete the registration;

[0010] S4. Image fusion: According to the registered images, convert the visible light image format from RGB to YUV, and superimpose the corresponding brightness values of the SAR image on the Y channel, calculate the local contrast and information entropy of the visible light image and the SAR image, dynamically adjust the fusion ratio of the visible light image and the SAR image, and generate a registered fusion image.

[0011] Preferably, the measurement and calibration method in step S1 specifically includes: gradually measuring the field of view angle of the visible light camera in the Y direction by using a folding visible SAR dual-light co-field optical tube; The folding visible SAR dual-light co-field optical tube contains a crosshair, and the stepping length is 0.1 m.

[0012] Preferably, the method for adjusting the field of view angle of a single pixel in the visible light image to match the field of view angle of a single pixel in the SAR image in step S2 is specifically as follows:

[0013] The airborne computer calculates the focal length of the visible light camera when the resolution of each pixel of the visible light camera is equal to the ground pixel resolution of the SAR image according to the flight altitude of the UAV and the azimuth and pitch angles of the optoelectronic payload ; The camera control board controls the visible light camera to adjust to the focal length , until the field of view angle of a single visible light pixel matches the field of view angle of a single SAR pixel.

[0014] Preferably, the heterogeneous image descriptors of the visible light image and the SAR image in step S3 include phase consistency descriptors and gradient descriptors; The method of weighted correlation distance is used for matching; The specific method is as follows:

[0015] S301. Calculate the phase consistency descriptor of the heterogeneous image to measure the phase of specific frequency components in the heterogeneous image; For each pixel, the phase can be calculated by the following formula:

[0016] ;

[0017] Among them, is the pixel coordinate, w is different scales, is the phase at scale w, and M represents the number of sizes;

[0018] S302. Calculate the gradient descriptors of heterogeneous images;

[0019] The gradient direction descriptors of heterogeneous images are as follows:

[0020] ; ;

[0021] Among them, and are the gradients in the x - direction and y - direction at the scale of m respectively, I is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as:

[0022]

[0023] ;

[0024] In the formula, represents the direction of the gradient, that is, the direction of the brightness change of the image at the point (x, y);

[0025] S303. Set the statistical scale as m, and the gradient histogram of 10×10 pixels centered on the (x, y) pixel is as follows:

[0026] ;

[0027] S304. Based on the size of the image target, take M / 2 scales upward and M / 2 scales downward, for a total of M scales; ; The phase - consistency descriptor 、gradient descriptor 、gradient histogram at the m - th scale form the descriptor vector at the m - th scale:

[0028] ;

[0029] S305. Use the method of weighted correlation distance for matching; the expression is as follows:

[0030] ;

[0031] In the formula, WCD represents the weighted correlation distance, that is, the weighted sum of the differences between the descriptor vectors of the visible - light image and the SAR image from scale 1 to scale M; represents the total multi - scale descriptor vector of the visible - light image; represents the total multi - scale descriptor vector of the SAR image; m is the scale of the descriptor; M represents the number of sizes; and respectively represent the multi-scale descriptor vectors at the m-th scale in the visible light image and the SAR image; is the weight of the m-th scale, expressed as: ; is the descriptor vector of the m-th scale and the variance of;

[0032] When the WCD value is greater than 0.5, it is considered that and match successfully; store the projection relationship of each pixel coordinate in the visible light image and the SAR image as matrix B.

[0033] Preferably, step S4 specifically includes the following sub-steps:

[0034] S401. Calculate the local contrast of the visible light image and the SAR image; normalize the local contrast of the visible light image and the SAR image to obtain the weights of the local contrast of the visible light image and the SAR image;

[0035] S402. Calculate the information entropy of the visible light image and the SAR image, normalize the information entropy of the visible light image and the SAR image to obtain the weights of the information entropy of the visible light image and the SAR image;

[0036] S403. Convert the RGB format file of the visible light image to YUV, and make the Y-channel brightness value Y ccd of the visible light image Y SAR superimpose with the Y-channel brightness value Y fused of the SAR image, and perform pixel-by-pixel fusion to generate a registered and fused image

[0037] ;

[0038] In the formula, Y fused represents the registered and fused image, Y ccd represents the Y-channel brightness value of the visible light image, Y SAR is the Y-channel brightness value of the SAR image; represents the weight of the visible light image based on the local contrast, represents the weight of the SAR image based on the local contrast; represents the weight of the visible light image based on the information entropy, represents the weight of the SAR image based on the information entropy.

[0039] Preferably, step S401 specifically includes the following sub-steps:

[0040] S4011. Determine the local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel values in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast;

[0041] The formulas for calculating the local standard deviation of visible light images and SAR images are as follows:

[0042] ;

[0043] ;

[0044] In the formula, is the pixel value of the visible image at the coordinate ([[]] x , y ), is the pixel value of the SAR image at the coordinate ([[]] x , y ), is the average pixel value of the local area Ω[[[]] f in the visible light image or SAR image, and ∣Ω[[[]] f ∣ represents the total number of pixels in the local area Ω[[[]] f in the visible light image or SAR image;

[0045] The expression for the local contrast of the visible light image or SAR image is as follows:

[0046] CCDLocalContrast = max([[]] );

[0047] ;

[0048] S4012. Normalize the local contrasts of the visible light image and the SAR image so that the sum of CCDLocalContrast and SARLocalContrast is 1; obtain the weights of the local contrasts of the visible light image and the SAR image:

[0049] ;

[0050] In the formula, represents the weight of the visible light image based on the local contrast, represents the weight of the SAR image based on the local contrast.

[0051] Preferably, step S402 specifically includes the following sub-steps:

[0052] S4021. Use a histogram to represent the number of times the gray value i appears in the visible light image or SAR image; for the gray value of each pixel point in the gray image or , perform the following operations:

[0053] ;

[0054] In the formula, and represent the visible histogram and the SAR histogram; i represents the gray value; represents the indicator function; when is equal to i, δ is 1, otherwise it is 0; when is equal to i, δ is 1, otherwise it is 0; represents the gray value of the pixel at coordinates (x, y) in the visible image, represents the gray value of the pixel at coordinates (x, y) in the SAR image;

[0055] S4022. Perform normalization to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image:

[0056] ;

[0057] ;

[0058] In the formula, and are the probability distributions of each gray value of the visible light image and the SAR image respectively, and are the width and height of the visible light image respectively, and are the width and height of the SAR image respectively;

[0059] S4023. Use the Shannon information entropy formula to multiply the probability of each gray value by its logarithm to the base 2 and sum over all 256 possible gray values to calculate the information entropy of the visible light image H ccd and the information entropy of the SAR image H SAR; The calculation formula is as follows:

[0060] ;

[0061] ;

[0062] S4024. Normalize the information entropy of the visible light image and the SAR image so that H ccd and H SAR sum to 1 to obtain the weights of the information entropy of the visible light image and the SAR image:

[0063] ;

[0064] In the formula, represents the weight of the visible light image based on information entropy, represents the weight of the SAR image based on information entropy.

[0065] Preferably, the value range of the information entropy is 0 to 8 bits; when all pixels have the same gray value, H ccd = 0, H SAR = 0; when the probability of each gray value appearing is equal, the information entropy H ccd = 8, H SAR = 8.

[0066] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0067] The present invention only needs to simply calibrate the field of view angles of visible light and SAR on the ground. After zooming, on the airborne computer, run the registration algorithm, and combine multi-scale descriptors and an optimized search strategy to improve the efficiency and accuracy of feature matching. It is applicable to the registration of visible SAR images under different resolutions and when the visible SAR radar is a zoom lens, especially applicable to the field of image registration such as the registration of a variable focal length visible camera and a variable ground resolution SAR radar in an integrated payload of an unmanned aerial vehicle (UAV) airborne optoelectronic radar, etc. It has a wide application prospect and practical application value, and significantly reduces the workload of registration calibration. Brief Description of the Drawings

[0068] Figure 1 is a flowchart of a method for registering and fusing visible light and SAR images during the dynamic flight of a UAV according to an embodiment of the present invention.

[0069] Figure 2 is a visible light image taken by a UAV aerial photography according to an embodiment of the present invention.

[0070] Figure 3 is a SAR image taken by a UAV aerial photography according to an embodiment of the present invention.

[0071] Figure 4 is a fused registration image generated after fusing the visible light image and the SAR image on a UAV according to an embodiment of the present invention. Detailed Embodiments

[0072] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same modules are denoted by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, their detailed descriptions will not be repeated.

[0073] To make the objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation to the present invention.

[0074] The present invention aims to solve the limitations of the prior art and provides a new image registration method. This method is applicable to the integrated payload system of an airborne optoelectronic radar of an unmanned aerial vehicle (UAV), can reduce the calibration workload, and improve the efficiency and accuracy of image registration. By performing simple field-of-view calibration on the ground and combining multi-scale descriptors and an optimized search strategy, the registration algorithm can be run in real time on an airborne computer to meet the requirements of real-time image processing of UAVs.

[0075] See Figure 1 , the present invention provides a method for registering and fusing visible light and SAR images during the dynamic flight of a UAV, which specifically includes the following steps:

[0076] S1. Ground calibration: Install the visible light camera on the airborne optoelectronic turret of the UAV; measure and calibrate the relationship between the focal length of the visible light camera and the field of view angle in the Y direction on the ground, record the relationship between the field of view angle and the focal length as a file, and store it on the camera control board;

[0077] The specific measurement and calibration method includes: using a folding visible SAR dual-optical co-field-of-view light pipe (the light pipe contains a crosshair, and the step length is 0.1 m) to gradually measure the field of view angle in the Y direction of the visible light camera;

[0078] Specifically, the azimuth and elevation encoders that have been measured are equipped inside the optoelectronic turret; assume that the focal length of the visible light camera lens at this time is CCD_F, the elevation angle of the optoelectronic turret is locked at 0, align the upper edge of the visible light camera with the crosshair of the light pipe, and read the azimuth angle CCDup of the optoelectronic turret at this time; rotate the optoelectronic turret so that the lower edge of the visible light camera aligns with the crosshair of the light pipe, and read the azimuth angle CCDdown of the optoelectronic turret at this time. Then, when the focal length of the visible light camera is CCD_F, the field of view angle in the Y direction of the visible light is CCD_Y = CCDup - CCDdown; record the corresponding relationship between the focal length of the visible light lens and the field of view angle into the array c.

[0079] S2. Coarse matching of ground pixel resolution: The camera control board controls the movement of the visible light camera through the PID algorithm until the field of view angle of a single pixel of the visible light image matches the field of view angle of a single pixel of the SAR image;

[0080] The specific operation is as follows: The airborne computer calculates the focal length of the visible light camera when the resolution of each pixel of the visible light camera is equal to the ground pixel resolution of the SAR image according to the flight altitude of the UAV and the azimuth and elevation angles of the optoelectronic payload ; the camera control board controls the visible light camera to adjust to the focal length through the PID algorithm until the field of view angle of a single visible light pixel matches that of a single SAR pixel.

[0081] If each pixel of the visible light image and the SAR image has the same ground pixel resolution, the derivation of the focal length relationship of the visible light lens is as follows: If the ground pixel resolution of the SAR image selected by the ground station is 0.2 m, it means that the ground size represented by each SAR pixel is 0.2 m × 0.2 m. The airborne computer calculates the focal length of the visible light camera when each pixel of the visible light camera represents a ground size of 0.2 m × 0.2 m based on the flight altitude of the UAV and the azimuth and pitch angles of the optoelectronic payload;

[0082] Considering the pitch angle θ the calculation formula for the focal length of the visible light camera is:

[0083] ;

[0084] In the formula, f is the focal length of the visible light camera; H is the flight altitude of the UAV; θ is the pitch angle of the optoelectronic payload, that is, the angle between the camera and the horizontal plane; S is the size of the visible light camera pixel; P is the ground size represented by the pixel, which is 0.2 meters here.

[0085] Similarly, calculate the focal length of the visible light camera when the SAR has other resolutions; the camera control board controls the visible light to adjust to the corresponding focal length through the PID algorithm and repeatedly adjusts until the field of view angle of a single visible light pixel matches that of a single SAR pixel.

[0086] S3. Image descriptor extraction and registration: Extract the heterogeneous image descriptors of the visible light image and the SAR image, match the descriptors, and store the projection relationship of each pixel coordinate of the visible light image and the SAR image as matrix B; call matrix B to complete the registration;

[0087] The heterogeneous image descriptors of the visible light image and the SAR image include phase consistency descriptors and gradient descriptors; the method of matching the descriptors is to combine the phase consistency descriptors and the gradient descriptors, extract the general feature descriptors of the visible light image and the SAR image, and use the method of weighted correlation distance (WCD) for matching; the specific method is as follows:

[0088] S301. Calculate the phase consistency descriptor of the heterogeneous image to measure the phase of specific frequency components in the heterogeneous image; for each pixel, the phase can be calculated by the following formula:

[0089] ;

[0090] Among them, is the pixel coordinate, w is different scales, is the phase at scale w, and M represents the number of sizes (usually taking the value of 10);

[0091] S302. Calculate the gradient descriptor of the heterogeneous image;

[0092] The gradient direction descriptor of the heterogeneous image is as follows:

[0093] ; ;

[0094] Among them, and are the gradients in the x - direction and y - direction at scale m respectively, I is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as:

[0095]

[0096] ;

[0097] In the formula, represents the direction of the gradient, indicating the direction of the brightness change of the image at the point (x, y);

[0098] S303. Set the statistical scale as m, and the gradient histogram of 10×10 pixels surrounding the center of the (x, y) pixel is expressed as follows:

[0099] ;

[0100] S304. Based on the target size of the image, take M / 2 scales upward and M / 2 scales downward, for a total of M scales. The phase consistency descriptor at the m - th scale , the gradient descriptor (gradient amplitude) , and the gradient histogram constitute the descriptor vector at the m - th scale:

[0101] ;

[0102] Both the visible - light image and the SAR image are operated according to the above - mentioned method, and a rich multi - scale descriptor vector and , this descriptor effectively describes the local structural information of visible images and SAR images, and provides a solid foundation for feature matching.

[0103] S305. Use the method of weighted correlation distance (WCD) for matching; the expression is as follows:

[0104] ;

[0105] In the formula, WCD represents the weighted correlation distance, that is, the weighted sum of the differences between the visible light image and the SAR image descriptor vectors from scale 1 to scale M; represents the total multi-scale descriptor vector of the visible light image, which contains information on all scales of the visible light; represents the total multi-scale descriptor vector of the SAR image, which contains information on all scales of the SAR image; m is the scale of the descriptor (the value range of m is from 1 to M); M represents the number of sizes (usually taking the value of 10); and respectively represent the multi-scale descriptor vectors at the m-th scale in the visible light image and the SAR image; is the weight of the m-th scale, expressed as: ; is the descriptor vector of the m-th scale and are the variances of;

[0106] When the WCD value is greater than 0.5, it is considered that and are successfully matched; the projection relationship of each pixel coordinate in the visible light image and the SAR image is stored as matrix B.

[0107] Principle brief: After step S2, it is theoretically ensured that the visible ground pixel resolution is basically the same as the SAR ground pixel resolution. However, due to the field of view angle calibration error in step S1 and the focal length control error in step S2, there are slight differences between the visible ground pixel resolution and the SAR ground pixel resolution. Therefore, registration operations are required. The calculation of the common feature descriptor of the visible light image and the SAR image is a key step in the matching of the visible light image and the SAR image, which provides a unique vector representation for each feature point of the visible light image and the SAR image.

[0108] Using the weighted correlation distance (WCD) method not only considers the Euclidean distance between feature vectors but also considers the correlation between feature dimensions. This method is particularly suitable for cases where there are internal connections between feature dimensions.

[0109] Because visible SAR uses external triggering to ensure that visible SAR is exposed at the same time, after exposure, a new frame of visible light image and SAR image arrive at the onboard computer. Only by calling matrix B can the real-time registration of visible SAR be completed, so that the real-time performance of the registration is effectively guaranteed.

[0110] S4. Image fusion: According to the registered image, convert the visible light image format from RGB to YUV, and superimpose the corresponding brightness value of the SAR image on the Y channel, calculate the local contrast and information entropy of the visible light image and the SAR image, dynamically adjust the fusion ratio of the visible light image and the SAR image, and generate a registered fused image; specifically, it includes the following sub-steps:

[0111] S401. Calculate the local contrast of the visible light image and the SAR image; normalize the local contrast of the visible light image and the SAR image to obtain the weight of the local contrast of the visible light image and the SAR image; specifically include the following sub-steps:

[0112] S4011. Determine a local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel values ​​in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast.

[0113] Local standard deviation It is a local area The measurement of the internal pixel value fluctuation, the local standard deviation calculation formulas for visible light images and SAR images are as follows:

[0114] ;

[0115] ;

[0116] In the formula, is the visible image at coordinates ( x , y ), is the SAR image at coordinates ( x , y ), is the local area Ω in the visible light image or SAR image f The average pixel value, |Ω f ∣ represents the local area Ω in the visible light image or SAR image f The total number of pixels.

[0117] The local standard deviation is calculated by calculating the local region Ω f Inner pixel value and its average value μ fThe sum of the squared deviations is then taken, and the square root is calculated. This value reflects the degree of dispersion of pixel intensities within the feature region and is an indicator of local texture complexity.

[0118] The local contrast of a visible light image or SAR image is defined as the maximum value of its local standard deviation:

[0119] CCDLocalContrast = max( ) ;

[0120] ;

[0121] S4012. Normalize the local contrasts of the visible light image and the SAR image so that the sum of CCDLocalContrast and SARLocalContrast is 1; obtain the weights of the local contrasts of the visible light image and the SAR image:

[0122] ;

[0123] In the formula, represents the weight of the visible light image based on local contrast, represents the weight of the SAR image based on local contrast.

[0124] S402. Calculate the information entropy of the visible light image and the SAR image, normalize the information entropy of the visible light image and the SAR image, and obtain the weights of the information entropy of the visible light image and the SAR image; specifically, it includes the following sub-steps:

[0125] S4021. Use a histogram to represent the number of times the gray value i appears in the visible light image or SAR image; for the gray value of each pixel point in the gray image or , perform the following operations:

[0126] ;

[0127] In the formula, and represent the visible histogram and the SAR histogram; i represents the gray value; represents the indicator function; when is equal to i, δ is 1, otherwise it is 0; when is equal to i, δ is 1, otherwise it is 0; represents the gray value of the pixel point with coordinates (x, y) in the visible image, represents the gray value of the pixel point with coordinates (x, y) in the SAR image.

[0128] S4022. Perform normalization to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image:

[0129] ;

[0130] ;

[0131] Where, and are the probability distributions of each gray value of the visible light image and the SAR image respectively, and are the width and height of the visible light image respectively, and are the width and height of the SAR image respectively.

[0132] S4023. Use the Shannon information entropy formula, multiply the probability of each gray value by its logarithm to the base 2, and accumulate over all 256 possible gray values to calculate the information entropy of the visible light image H ccd and the information entropy of the SAR image H SAR;

[0133] ;

[0134] ;

[0135] The range of the information entropy is from 0 to the maximum value of 8 bits, and this range reflects the amount of information in the image from completely disordered (the gray value of each pixel point is random and has equal probability) to completely ordered (all pixel points have the same gray value); when all pixels have the same gray value (i.e., the image is completely uniform), H ccd = 0, H SAR = 0; when the probability of each gray value appearing is equal, the information entropy H ccd = 8, H SAR = 8, reaching the maximum value.

[0136] S4024. Normalize the information entropy of the visible light image and the SAR image so that H ccd and H SAR sum to 1 to obtain the weights of the information entropy of the visible light image and the SAR image:

[0137] ;

[0138] Where, represents the weight of the visible light image based on the information entropy, represents the weight of the SAR image based on the information entropy.

[0139] S403. Convert the RGB format file of the visible light image to YUV, and make the luminance value of the Y channel of the visible light image according to matrix B Y The luminance values of the Y channels of the ccd and SAR images Y SAR Overlay them, perform fusion pixel by pixel, and generate a registered and fused image Y fused, the expression is as follows:

[0140] ;

[0141] In the formula, Y fused represents the registered and fused image, Y ccd represents the luminance value of the Y channel of the visible light image, Y SAR is the luminance value of the Y channel of the SAR image; represents the weight of the visible light image based on local contrast, represents the weight of the SAR image based on local contrast; represents the weight of the visible light image based on information entropy, represents the weight of the SAR image based on information entropy.

[0142] In summary, the present invention simplifies the ground calibration process. It only needs to calibrate the field of view angles and imaging parameters of the visible light camera and the SAR. After zooming or parameter changes, run the registration algorithm on the airborne computer, extract the heterogeneous image descriptors, match the descriptors of the visible light and SAR images, and store the projection relationship of each pixel coordinate of the visible light and SAR as matrix B. When the focal length or parameter configuration remains unchanged, this matrix B only needs to be calculated once and stored in the airborne computer.

[0143] Since the visible light and the SAR adopt a synchronous triggering method to ensure that the visible light and the SAR acquire data at the same time. After the data is acquired, a new frame of visible light image and SAR image arrive at the airborne computer simultaneously. Just call matrix B to complete the real-time registration of the visible light and the SAR, effectively ensuring the real-time performance of the registration. After each frame of image is registered through matrix B, convert the visible light video format from RGB to YUV, and overlay the corresponding luminance value of the SAR on the Y channel, calculate the local contrast and information entropy of the visible light and the SAR, and dynamically adjust the fusion ratio of the visible light Y and SAR luminance values according to the local contrast and information entropy to maximize the information content of the fused image.

[0144] The method of the present invention is applicable to image registration under different resolutions and variable focal lengths or variable parameter configurations of visible light cameras and SARs, and is particularly applicable to fields such as image registration of variable focal length visible light cameras and SARs of integrated optoelectronic radar payloads on unmanned aerial vehicles. It has broad application prospects and practical application value, and significantly reduces the workload of registration and calibration.

[0145] It should be understood that various forms of processes shown above can be used, with steps reordered, added or deleted. For example, the steps described in the disclosure of the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the present invention can be achieved. No limitations are imposed herein.

[0146] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub - combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for registering and fusing visible light and SAR images in dynamic flight of unmanned aerial vehicles, characterized in that: The specific steps include: S1. Ground calibration: measure and calibrate the relationship between the focal length of the visible light camera and the field of view angle in the Y direction of the visible light on the ground, record the relationship between the field of view angle and the focal length as a file, and store it on the camera control board; S2. Rough matching of ground pixel resolution: The camera control board controls the movement of the visible light camera through the PID algorithm until the field of view angle of a single pixel in the visible light image matches the field of view angle of a single pixel in the SAR image; S3. Image descriptor extraction and registration: extract heterogeneous image descriptors of visible light image and SAR image, match the descriptors, store the projection relationship of each pixel coordinate of visible light image and SAR image as matrix B; call matrix B to complete the registration; S4. Image fusion: According to the registered image, convert the visible light image format from RGB to YUV, and superimpose the corresponding brightness value of the SAR image on the Y channel, calculate the local contrast and information entropy of the visible light image and the SAR image, dynamically adjust the fusion ratio of the visible light image and the SAR image, and generate a registered fused image; specifically, it includes the following sub-steps: S401. Calculate the local contrast of the visible light image and the SAR image; normalize the local contrast of the visible light image and the SAR image to obtain the weight of the local contrast of the visible light image and the SAR image; S402. Calculate the information entropy of the visible light image and the information entropy of the SAR image, normalize the information entropy of the visible light image and the SAR image, and obtain the weights of the information entropy of the visible light image and the SAR image; S403. Convert the RGB format file of the visible light image to YUV, and adjust the brightness value of the Y channel of the visible light image according to matrix B. Y Y channel brightness value of CCD and SAR images Y SAR Overlay and fuse pixel by pixel to generate a registered fused image Y fused, the expression is as follows: ； In the formula, Y fused means to register the fused image. Y CCD represents the Y channel brightness value of the visible light image. Y SAR is the brightness value of the Y channel of the SAR image; represents the visible light image weight based on local contrast, represents the SAR image weight based on local contrast; represents the weight of the visible light image based on information entropy, Represents the SAR image weight based on information entropy.

2. The method for registration and fusion of visible light and SAR images in dynamic flight of unmanned aerial vehicle according to claim 1, characterized in that: The measurement and calibration method in step S1 specifically includes: using a reentrant visible SAR dual-light common-field light pipe to gradually measure the Y-direction field of view angle of the visible light camera; the reentrant visible SAR dual-light common-field light pipe contains a crosshair, and the step length is 0.1m.

3. The method for registration and fusion of visible light and SAR images in dynamic flight of unmanned aerial vehicle according to claim 1, characterized in that: The method for adjusting the field of view angle of a single pixel of the visible light image to match the field of view angle of a single pixel of the SAR image in step S2 is specifically as follows: The onboard computer calculates the focal length of the visible light camera when the resolution of each pixel of the visible light camera is equal to the ground pixel resolution of the SAR image based on the UAV flight altitude and the azimuth and pitch angle of the optoelectronic payload. ; The camera control board uses the PID algorithm to control the visible light camera to adjust to the focal length , until the field of view of a single visible light pixel matches the field of view of a single SAR pixel.

4. The method for registration and fusion of visible light and SAR images in dynamic flight of unmanned aerial vehicle according to claim 1, characterized in that: In step S3, the heterogeneous image descriptors of the visible light image and the SAR image include a phase consistency descriptor and a gradient descriptor; the weighted correlation distance method is used for matching; the specific method is as follows: S301. Calculate the phase consistency descriptor of the heterogeneous image, which is used to measure the phase of a specific frequency component in the heterogeneous image; for each pixel, the phase is calculated by the following formula: ； in, are pixel coordinates, w are different scales, is the phase at scale w, M represents the number of scales; S302. Calculate the gradient descriptor of the heterogeneous image; The gradient direction descriptor of the heterogeneous image is as follows: ； ； in, and They are the gradients in the x and y directions when the scale is m, I is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as: ； In the formula, Indicates the direction of the gradient, that is, the direction of the brightness change of the image at the point (x, y); S303. Set the statistical scale to m, and take the pixel (x, y) as the center, and the gradient histogram of the surrounding 10×10 pixels is expressed as follows: ； S304. Based on the image target size, take M / 2 scales upward and M / 2 scales downward, for a total of M scales; ; Phase consistency descriptor at the mth scale , gradient descriptor , gradient histogram It constitutes the descriptor vector at the mth scale : ； S305. Use the weighted correlation distance method to perform matching; the expression is as follows: ； Where WCD represents the weighted correlation distance, which is the weighted sum of the differences between the descriptor vectors of the visible light image and the SAR image from scale 1 to scale M; Represents the total multi-scale descriptor vector of the visible light image; represents the total multi-scale descriptor vector of the SAR image; m is the scale of the descriptor; M represents the number of scales; and Represent the multi-scale descriptor vector at the mth scale in the visible light image and SAR image respectively; is the weight of the mth scale, expressed as: ; is the descriptor vector of the mth scale and The variance of When the WCD value is greater than 0.5, it is considered and The match is successful; the projection relationship between the visible light image and each pixel coordinate in the SAR image is stored as a matrix B.

5. The method for registration and fusion of visible light and SAR images in dynamic flight of unmanned aerial vehicle according to claim 1, characterized in that: The step S401 specifically includes the following sub-steps: S4011. Determine a local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel value in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast; The calculation formulas for the local standard deviation of visible light images and SAR images are as follows: ； ； In the formula, is the visible image at coordinates ( x , y ), is the SAR image at coordinates ( x , y ), is the local area Ω in the visible light image or SAR image f The average pixel value, |Ω f ∣ represents the local area Ω in the visible light image or SAR image f The total number of pixels; The local contrast expression of visible light image or SAR image is as follows: CCDLocalContrast=max( ); ； S4012. Normalize the local contrast of the visible light image and the SAR image so that the sum of CCDLocalContrast and SARLocalContrast is 1; obtain the weight of the local contrast of the visible light image and the SAR image: ； In the formula, represents the visible light image weight based on local contrast, Represents the SAR image weight based on local contrast.

6. The method for registration and fusion of visible light and SAR images in dynamic flight of unmanned aerial vehicle according to claim 5, characterized in that: The step S402 specifically includes the following sub-steps: S4021. Use a histogram to represent the number of times the gray value i appears in the visible light image or SAR image; for each pixel in the gray image, the gray value or , do the following: ； In the formula, and Represents visible histogram and SAR histogram; i represents gray value; represents the indicator function; when When it is equal to i, δ is 1, otherwise it is 0; when When it is equal to i, δ is 1, otherwise it is 0; Represents the grayscale value of the pixel with coordinates (x, y) in the visible image. Represents the grayscale value of the pixel with coordinates (x, y) in the SAR image; S4022. Perform normalization to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image: ； ； In the formula, and are the probability distribution of each gray value of the visible light image and SAR image, and are the width and height of the visible light image, respectively. and are the width and height of the SAR image respectively; S4023. Using the Shannon information entropy formula, multiply the probability of each gray value by its logarithm with base 2, and accumulate all 256 possible gray values ​​to calculate the information entropy of the visible light image. H Information entropy of CCD and SAR images H SAR; the calculation formula is as follows: ； ； S4024. Normalize the information entropy of the visible light image and the SAR image so that H CCD and H The sum of SAR is 1, and the weights of the information entropy of the visible light image and the SAR image are obtained: ； In the formula, represents the weight of the visible light image based on information entropy, Represents the SAR image weight based on information entropy.

7. The method for registration and fusion of visible light and SAR images in dynamic flight of unmanned aerial vehicle according to claim 6, characterized in that: The value range of the information entropy is 0 to 8 bits; when all pixels have the same grayscale value, H CCD = 0, H SAR=0; when the probability of each gray value appearing is equal, the information entropy H CCD = 8, H SAR=8.

Images