Method for registering and fusing infrared image and SAR image in dynamic flight of unmanned aerial vehicle

Abstract

The invention relates to the technical field of unmanned aerial vehicle load system image registration, in particular to an infrared image and SAR image registration fusion method in dynamic flight of an unmanned aerial vehicle. Comprising the following steps: measuring and calibrating a relationship between an infrared camera focal length and an infrared Y-direction field angle on the ground, recording the relationship between the field angle and the focal length as a file, and storing the file on a camera control panel; the camera control panel controls the movement of the infrared camera through a PID algorithm until the field angle of the single pixel of the infrared image is matched with the field angle of the single pixel of the SAR image; extracting different-source image descriptors, matching the different-source image descriptors, storing the projection relation of each pixel coordinate as a matrix C, and calling to complete registration; and calculating the local contrast and information entropy of the infrared image and the SAR image, dynamically adjusting the fusion proportion, and generating a registration fusion image. The method has the advantages that simple calibration on the ground is only needed; and after zooming, a registration algorithm is operated on an airborne computer, and the efficiency and accuracy of feature matching are improved in combination with a multi-scale descriptor.

Description

Technical Field

[0001] The invention relates to the technical field of image registration of unmanned aerial vehicle payload systems, and in particular to a method for registration and fusion of infrared images and SAR images in dynamic flight of unmanned aerial vehicles. Background Art

[0002] Traditional image registration methods usually require a complex ground calibration process, which includes precise measurement of the center offset of the visual axis, edge distortion, and pixel mapping relationship of infrared and SAR radars. These calibration processes are often time-consuming and labor-intensive, and are only applicable to fixed-focal-length lenses. However, in the integrated payload system of UAV-mounted optoelectronic radar, the infrared is equipped with a zoom lens, and the SAR radar usually has an adjustable ground resolution, which makes the traditional calibration method no longer applicable. In addition, due to the dynamic nature of the UAV platform and the real-time requirements of the mission, traditional image registration methods face challenges in practical applications. When the UAV is performing monitoring tasks, changes in the focal length of the camera are normal, which requires the image registration algorithm to quickly adapt to changes in the infrared focal length and changes in the ground resolution of the SAR radar to achieve real-time processing.

[0003] In the prior art, infrared image and SAR image registration usually requires the execution of cumbersome ground calibration procedures, including calibration of infrared and SAR visual axis center offset, imaging mode difference, pixel mapping relationship, etc. The calibration process is complicated and only applicable to the case where the infrared lens is a fixed focal length lens and the SAR is a fixed parameter configuration. The infrared camera of the photoelectric radar integrated payload is a variable focal length, and the SAR system is a variable parameter configuration. The calibration workload of center offset, imaging mode adaptation, pixel mapping, etc. for each focal length or parameter configuration is huge.

[0004] The main limitations of existing technologies are their complex ground calibration process and the lack of image registration solutions suitable for zoom lenses; there is a lack of algorithms that can adapt to the focal length changes of the variable-focal-length infrared camera and the variable-ground-resolution SAR radar of the UAV-mounted optoelectronic radar integrated payload and achieve fast and accurate registration. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a method for registering and fusing infrared images and SAR images during the dynamic flight of a UAV.

[0006] The present invention aims to provide a method for registering and fusing infrared images and SAR images in dynamic flight of unmanned aerial vehicles, which specifically comprises the following steps: S1. Ground calibration: measure and calibrate the relationship between the focal length of the infrared camera and the infrared Y-direction field of view angle on the ground, record the relationship between the field of view angle and the focal length as a file, and store it on the camera control board; S2. Rough matching of ground pixel resolution: The camera control board controls the movement of the infrared camera through the PID algorithm until the field of view angle of a single pixel in the infrared image matches the field of view angle of a single pixel in the SAR image; S3. Image descriptor extraction and registration: Extract heterogeneous image descriptors of infrared images and SAR images, match the descriptors, store the projection relationship of each pixel coordinate of the infrared image and the SAR image as a matrix C; call the matrix C to complete the registration; S4. Image fusion: According to the registered image, the infrared image format is converted from RGB to YUV, and the corresponding brightness value of the SAR image is superimposed on the Y channel. The local contrast and information entropy of the infrared image and SAR image are calculated, and the fusion ratio of the infrared image and SAR image is dynamically adjusted to generate a registered fused image.

[0007] Preferably, the measurement and calibration method in step S1 specifically includes: using a reentrant infrared SAR dual-light common-field light pipe to gradually measure the Y-direction field of view angle of the infrared camera; the reentrant infrared SAR dual-light common-field light pipe contains a crosshair, and the step length is 0.1m.

[0008] Preferably, the method for adjusting the field of view angle of a single pixel of the infrared image to match the field of view angle of a single pixel of the SAR image in step S2 is specifically as follows: The onboard computer calculates the focal length of the infrared camera when each pixel of the infrared camera has the same resolution as the ground pixel of the SAR image based on the flight altitude of the UAV and the azimuth and pitch angle of the optoelectronic payload. ; The camera control board uses the PID algorithm to control the infrared camera to adjust to the focal length , until the field of view of a single infrared pixel matches the field of view of a single SAR pixel.

[0009] Preferably, in step S3, the heterogeneous image descriptors of the infrared image and the SAR image include a phase consistency descriptor and a gradient descriptor; and the weighted correlation distance method is used for matching; the specific method is as follows: S301. Calculate the phase consistency descriptor of the heterogeneous image, which is used to measure the phase of a specific frequency component in the heterogeneous image; for each pixel, the phase can be calculated by the following formula: ; in, are pixel coordinates, w are different scales, is the phase at scale w, M represents the number of sizes; S302. Calculate the gradient descriptor of the heterogeneous image; The gradient direction descriptor of the heterogeneous image is as follows: ; ; in, and are the gradients in the x and y directions when the scale is m, respectively, and 𝐼 is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as:

[0010] ; S303. Set the statistical scale to m, and take the (x, y) pixel as the center and the gradient histogram of the surrounding 10×10 pixels as follows: ; S304. Based on the image target size, take M / 2 scales upward and M / 2 scales downward, for a total of M scales; ; Phase consistency descriptor at the mth scale , gradient descriptor , gradient histogram It constitutes the descriptor vector at the mth scale : ; S305. Use the weighted correlation distance method to perform matching; the expression is as follows: ; Where WCD represents the weighted correlation distance, which is the weighted sum of the differences between the infrared image and the SAR image descriptor vectors from scale 1 to scale M; m is the scale of the descriptor; M represents the number of sizes; and Respectively represent the multi-scale descriptor vector of each feature point in the infrared image and the SAR image; is the weight of the mth scale, expressed as: ; is the descriptor vector of the mth scale and The variance of When the WCD value is greater than 0.5, it is considered and The match is successful; the projection relationship between the infrared image and each pixel coordinate in the SAR image is stored as a matrix C.

[0011] Preferably, step S4 specifically includes the following sub-steps: S401. Calculate the local contrast of the infrared image and the SAR image; normalize the local contrast of the infrared image and the SAR image to obtain the weight of the local contrast of the infrared image and the SAR image; S402. Calculate the information entropy of the infrared image and the information entropy of the SAR image, normalize the information entropy of the infrared image and the SAR image, and obtain the weights of the information entropy of the infrared image and the SAR image; S403. Convert the RGB format file of the infrared image to YUV, and make the Y channel brightness value of the infrared image Y ir and Y channel brightness value of SAR image Y SAR Overlay and fuse pixel by pixel to generate a registered fused image Y fused, the expression is as follows: ; In the formula, Y fused means to register the fused image. Y ir represents the Y channel brightness value of the infrared image, Y SAR is the Y channel brightness value of the SAR image; represents the infrared image weight based on local contrast, represents the SAR image weight based on local contrast; represents the infrared image weight based on information entropy, Represents the SAR image weight based on information entropy.

[0012] Preferably, step S401 specifically includes the following sub-steps: S4011. Determine a local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel value in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast; The calculation formulas for the local standard deviation of infrared images and SAR images are as follows: ; ; In the formula, is the infrared image at coordinates ( x , y ), is the SAR image at coordinates ( x , y ), is the local area Ω in the infrared image or SAR image f The average pixel value, |Ω f ∣ represents the local area Ω in the infrared image or SAR image f The total number of pixels; The local contrast expression of infrared image or SAR image is as follows: ; ; S4012. Normalize the local contrast of the infrared image and the SAR image so that the sum of IRLocalContrast and SARLocalContrast is 1; obtain the weight of the local contrast of the infrared image and the SAR image: ; In the formula, represents the infrared image weight based on local contrast, Represents the SAR image weight based on local contrast.

[0013] Preferably, step S402 specifically includes the following sub-steps: S4021. Use a histogram to represent the number of times the gray value 𝑖 appears in the infrared image or SAR image; for each pixel in the gray image, the gray value or , do the following: ; In the formula, and Indicates infrared histogram and SAR histogram; i indicates gray value; represents the indicator function; when When it is equal to i, is 1, otherwise it is 0; When it is equal to i, is 1, otherwise it is 0; Represents the grayscale value of the pixel with coordinates (x, y) in the infrared image. Represents the grayscale value of the pixel with coordinates (x, y) in the SAR image; S4022. Perform normalization to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image: ; ; In the formula, and are the probability distribution of each gray value of infrared image and SAR image respectively, and are the width and height of the infrared image, and are the width and height of the SAR image respectively; S4023. Use the Shannon information entropy formula to multiply the probability of each gray value by its logarithm with base 2, and accumulate all 256 possible gray values ​​to calculate the infrared image information entropy 𝐻ir and the SAR image information entropy 𝐻SAR; the calculation formula is as follows: ; ; S4024. Normalize the information entropy of the infrared image and the SAR image so that H ir and H The sum of SAR is 1, and the weights of the information entropy of the infrared image and the SAR image are obtained: ; In the formula, represents the infrared image weight based on information entropy, Represents the SAR image weight based on information entropy.

[0014] Preferably, the value range of information entropy is 0 to 8 bits; when all pixels have the same grayscale value, H ir=0, H SAR=0; when the probability of each gray value appearing is equal, the information entropy H ir=8, H SAR=8.

[0015] Compared with the prior art, the present invention can achieve the following beneficial effects: The present invention only needs to simply calibrate the field of view of infrared and SAR on the ground, and after zooming, run the registration algorithm on the onboard computer, combining multi-scale descriptors and optimized search strategies to improve the efficiency and accuracy of feature matching. It is suitable for infrared SAR image registration under the condition of different resolutions and infrared SAR radar as a zoom lens, and is particularly suitable for image registration under the conditions of variable focal length infrared camera with integrated load of photoelectric radar on unmanned aerial vehicle and variable ground resolution SAR radar, etc. It has broad application prospects and practical application value, and significantly reduces the workload of registration and calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The present invention provides a flowchart of a method for registering and fusing infrared images and SAR images during the dynamic flight of a UAV according to an embodiment of the present invention.

[0017] Figure 2 This is an infrared image taken by a drone according to an embodiment of the present invention.

[0018] Figure 3 This is a SAR image taken by a drone according to an embodiment of the present invention.

[0019] Figure 4 It is a fused registration image generated by fusing the infrared image and the SAR image on the drone provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0021] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0022] The present invention aims to solve the limitations of the prior art and provide a new image registration method, which is applicable to the integrated payload system of the photoelectric radar onboard the UAV, can reduce the calibration workload and improve the efficiency and accuracy of image registration. By performing a simple field of view angle calibration on the ground, combined with multi-scale descriptors and optimized search strategies, the registration algorithm can be run in real time on the onboard computer to meet the needs of real-time image processing of the UAV.

[0023] See also Figure 1 The present invention provides a method for registering and fusing infrared images and SAR images in dynamic flight of a UAV, which specifically comprises the following steps: S1. Ground calibration: Install the infrared camera on the optoelectronic turret on the drone; measure and calibrate the relationship between the focal length of the infrared camera and the infrared Y-direction field of view on the ground, record the relationship between the field of view and the focal length as a file, and store it on the camera control board; The measurement and calibration method specifically includes: using a reentrant infrared SAR dual-light common field light tube (the light tube contains crosshairs, and the step length is 0.1m) to gradually measure the Y-direction field of view angle of the infrared camera; Specifically, the inside of the optoelectronic turret is equipped with a measured azimuth and pitch encoder; assuming that the focal length of the infrared camera lens is IR_F at this time, the pitch angle of the optoelectronic turret is locked to 0, align the upper edge of the infrared camera with the crosshairs of the light tube, and read the azimuth angle IRup of the optoelectronic turret at this time; rotate the optoelectronic turret to align the lower edge of the infrared camera with the crosshairs of the light tube, and read the azimuth angle IRdown of the optoelectronic turret at this time, then when the focal length of the infrared camera is IR_F, the infrared Y direction field of view angle is IR_Y=IRup-IRdown; record the correspondence between the focal length of the infrared lens and the field of view angle into array c.

[0024] S2. Rough matching of ground pixel resolution: The camera control board controls the movement of the infrared camera through the PID algorithm until the field of view angle of a single pixel in the infrared image matches the field of view angle of a single pixel in the SAR image; The specific operation is as follows: The onboard computer calculates the focal length of the infrared camera when each pixel of the infrared camera has the same resolution as the ground pixel of the SAR image based on the flight altitude of the UAV and the azimuth and pitch angle of the optoelectronic payload. ; The camera control board uses the PID algorithm to control the infrared camera to adjust to the focal length , until the field of view of a single infrared pixel matches the field of view of a single SAR pixel.

[0025] Each pixel of the infrared image and the SAR image has the same ground pixel resolution, so the relationship of the infrared lens focal length is derived as follows: If the ground pixel resolution of the SAR image selected by the ground station is 0.2m, it means that each SAR pixel represents a ground size of 0.2m×0.2m. The onboard computer calculates the infrared camera focal length when each infrared camera pixel represents a ground size of 0.2m×0.2m based on the UAV flight altitude and the azimuth and pitch angle of the optoelectronic payload; Consider the pitch angle θ In the case of , the calculation formula for the focal length of the infrared camera is: ; In the formula, f is the focal length of the infrared camera; H is the flight altitude of the drone; θ is the pitch angle of the photoelectric payload, that is, the angle of the camera relative to the horizontal plane; S is the size of the infrared camera pixel; P is the ground size represented by the pixel, which is 0.2 meters here.

[0026] Similarly, calculate the focal length of the infrared camera when the SAR is at other resolutions; the camera control board controls the infrared to adjust to the corresponding focal length through the PID algorithm , repeatedly adjust until the field of view of a single infrared pixel matches the field of view of a single SAR pixel.

[0027] S3. Image descriptor extraction and registration: Extract heterogeneous image descriptors of infrared images and SAR images, match the descriptors, store the projection relationship of each pixel coordinate of the infrared image and the SAR image as a matrix C; call the matrix C to complete the registration; The heterogeneous image descriptors of infrared images and SAR images include phase consistency descriptors and gradient descriptors. The method for matching descriptors is to extract the common feature descriptors of infrared images and SAR images by combining the phase consistency descriptors and gradient descriptors, and then use the weighted correlation distance (WCD) method for matching. The specific method is as follows: S301. Calculate the phase consistency descriptor of the heterogeneous image, which is used to measure the phase of a specific frequency component in the heterogeneous image; for each pixel, the phase can be calculated by the following formula: ; in, are pixel coordinates, w are different scales, is the phase at scale w, M represents the number of sizes (usually 10); S302. Calculate the gradient descriptor of the heterogeneous image; The gradient direction descriptor of the heterogeneous image is as follows: ; ; in, and are the gradients in the x and y directions when the scale is m, respectively, and 𝐼 is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as:

[0028] ; S303. Set the statistical scale to m, and take the (x, y) pixel as the center and the gradient histogram of the surrounding 10×10 pixels as follows: ; S304. Based on the image target size, take M / 2 scales upward and M / 2 scales downward, for a total of M scales. Phase consistency descriptor under , gradient descriptor (gradient magnitude) , gradient histogram It constitutes the descriptor vector at the mth scale : ; The infrared image and SAR image are operated according to the above method, and each feature point in the infrared image and SAR image generates a rich multi-scale descriptor vector and ,This descriptor effectively describes the local structural information of infrared images and SAR images, and provides a solid foundation for feature matching.

[0029] S305. Use the weighted correlation distance (WCD) method for matching; the expression is as follows: ; Where WCD represents the weighted correlation distance, which is the weighted sum of the differences between the infrared image and the SAR image descriptor vectors from scale 1 to scale M; m is the scale of the descriptor (the value of m ranges from 1 to M); M represents the number of sizes (usually 10); and Respectively represent the multi-scale descriptor vector of each feature point in the infrared image and the SAR image; is the weight of the mth scale, expressed as: ; is the descriptor vector of the mth scale and The variance of When the WCD value is greater than 0.5, it is considered and The match is successful; the projection relationship between the infrared image and each pixel coordinate in the SAR image is stored as a matrix C.

[0030] Principle: After step S2, the infrared ground pixel resolution is basically consistent with the SAR ground pixel resolution in theory. However, due to the field of view calibration error in step S1 and the focal length control error in step S2, there is a slight difference between the infrared ground pixel resolution and the SAR ground pixel resolution. Therefore, a registration operation is required. The calculation of the universal feature descriptor of the infrared image and the SAR image is a key step in the matching of the infrared image and the SAR image, which provides a unique vector representation for each feature point of the infrared image and the SAR image.

[0031] The weighted correlation distance (WCD) method not only considers the Euclidean distance between feature vectors, but also the correlation between feature dimensions. This method is particularly suitable for situations where there are intrinsic connections between feature dimensions.

[0032] Because infrared SAR uses external triggering to ensure that infrared SAR is exposed at the same time, after exposure, a new frame of infrared image and SAR image arrive at the onboard computer. Only by calling matrix C can the infrared SAR real-time registration be completed, so that the real-time performance of the registration is effectively guaranteed.

[0033] S4. Image fusion: According to the registered image, the infrared image format is converted from RGB to YUV, and the corresponding brightness value of the SAR image is superimposed on the Y channel, the local contrast and information entropy of the infrared image and the SAR image are calculated, and the fusion ratio of the infrared image and the SAR image is dynamically adjusted to generate a registered fused image; specifically, the following sub-steps are included: S401. Calculate the local contrast of the infrared image and the SAR image; normalize the local contrast of the infrared image and the SAR image to obtain the weight of the local contrast of the infrared image and the SAR image; specifically include the following sub-steps: S4011. Determine a local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel values ​​in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast.

[0034] Local standard deviation It is a local area The measurement of internal pixel value fluctuation, the local standard deviation calculation formulas of infrared images and SAR images are as follows: ; ; In the formula, is the infrared image at coordinates ( x , y ), is the SAR image at coordinates ( x , y ), is the local area Ω in the infrared image or SAR image f The average pixel value, |Ω f ∣ represents the local area Ω in the infrared image or SAR image f The total number of pixels.

[0035] The local standard deviation is calculated by calculating the local region Ω f Inner pixel value and its average value μf The sum of the squared deviations of , and then take the square root. This value reflects the dispersion of pixel intensity in the feature area and is an indicator of the complexity of local texture.

[0036] The local contrast of an infrared image or SAR image is defined as the maximum value of its local standard deviation: ; ; S4012. Normalize the local contrast of the infrared image and the SAR image so that the sum of IRLocalContrast and SARLocalContrast is 1; obtain the weight of the local contrast of the infrared image and the SAR image: ; In the formula, represents the infrared image weight based on local contrast, Represents the SAR image weight based on local contrast.

[0037] S402. Calculate the information entropy of the infrared image and the information entropy of the SAR image, normalize the information entropy of the infrared image and the SAR image, and obtain the weights of the information entropy of the infrared image and the SAR image; specifically, the following sub-steps are included: S4021. Use a histogram to represent the number of times the gray value 𝑖 appears in the infrared image or SAR image; for each pixel in the gray image, the gray value or , do the following: ; In the formula, and Indicates infrared histogram and SAR histogram; i indicates gray value; represents the indicator function; when When it is equal to i, is 1, otherwise it is 0; When it is equal to i, is 1, otherwise it is 0; Represents the grayscale value of the pixel with coordinates (x, y) in the infrared image. Represents the grayscale value of the pixel with coordinates (x, y) in the SAR image.

[0038] S4022. Perform normalization to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image: ; ; In the formula, and are the probability distribution of each gray value of infrared image and SAR image respectively, and are the width and height of the infrared image, and are the width and height of the SAR image respectively.

[0039] S4023. Use the Shannon information entropy formula to multiply the probability of each gray value by its logarithm with base 2, and accumulate all 256 possible gray values ​​to calculate the infrared image information entropy 𝐻ir and the SAR image information entropy 𝐻SAR; ; ; The range of information entropy is from 0 to a maximum of 8 bits. This range reflects the amount of information in an image from complete disorder (the grayscale value of each pixel is random and has equal probability) to complete order (all pixels have the same grayscale value); when all pixels have the same grayscale value (that is, the image is completely uniform), H ir=0, H SAR=0; when the probability of each gray value appearing is equal, the information entropy H ir=8, H SAR=8, reaching the maximum value.

[0040] S4024. Normalize the information entropy of the infrared image and the SAR image so that H ir and H The sum of SAR is 1, and the weights of the information entropy of the infrared image and the SAR image are obtained: ; In the formula, represents the infrared image weight based on information entropy, Represents the SAR image weight based on information entropy.

[0041] S403. Convert the RGB format file of the infrared image to YUV, and make the Y channel brightness value of the infrared image Y ir and Y channel brightness value of SAR image Y SAR Overlay and fuse pixel by pixel to generate a registered fused image Y fused, the expression is as follows: ; In the formula, Y fused means to register the fused image. Y ir represents the Y channel brightness value of the infrared image, Y SAR is the Y channel brightness value of the SAR image; represents the infrared image weight based on local contrast, represents the SAR image weight based on local contrast; represents the infrared image weight based on information entropy, Represents the SAR image weight based on information entropy.

[0042] In summary, the present invention simplifies the ground calibration process and only needs to calibrate the field of view and imaging parameters of the infrared camera and SAR. After the focal length or parameter changes, the registration algorithm is run on the airborne computer to extract the heterogeneous image descriptors, match the descriptors of the infrared image and the SAR image, and store the projection relationship between the infrared and SAR pixel coordinates as a matrix C. When the focal length or parameter configuration remains unchanged, the matrix C only needs to be calculated once and stored in the airborne computer.

[0043] Because infrared and SAR use synchronous triggering to ensure that infrared and SAR acquire data at the same time, after data acquisition, a new frame of infrared image and SAR image arrive at the onboard computer at the same time, and only need to call matrix C to complete the real-time registration of infrared and SAR, so that the real-time registration is effectively guaranteed. After each frame of image is registered by matrix C, the infrared video format is converted from RGB to YUV, and the corresponding brightness value of SAR is superimposed on the Y channel, and the local contrast and information entropy of infrared and SAR are calculated. According to the local contrast and information entropy, the fusion ratio of infrared Y and SAR brightness values ​​is dynamically adjusted to maximize the information content of the fused image.

[0044] The method of the present invention is suitable for image registration under conditions of different resolutions and when the infrared camera and SAR are in variable focal length or variable parameter configurations, and is particularly suitable for the fields of image registration of variable focal length infrared camera and SAR with integrated load of photoelectric radar onboard unmanned aerial vehicles, etc. It has broad application prospects and practical application value, and significantly reduces the workload of registration and calibration.

[0045] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0046] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for registering and fusing infrared images and SAR images during dynamic flight of unmanned aerial vehicles, characterized in that: The specific steps include: S1. Ground calibration: measure and calibrate the relationship between the focal length of the infrared camera and the infrared Y-direction field of view angle on the ground, record the relationship between the field of view angle and the focal length as a file, and store it on the camera control board; S2. Rough matching of ground pixel resolution: The camera control board controls the movement of the infrared camera through the PID algorithm until the field of view angle of a single pixel in the infrared image matches the field of view angle of a single pixel in the SAR image; S3. Image descriptor extraction and registration: Extract heterogeneous image descriptors of infrared images and SAR images, match the descriptors, and store the projection relationship of each pixel coordinate of the infrared image and the SAR image as a matrix C; Call matrix C to complete the registration; S4. Image fusion: According to the registered image, the infrared image format is converted from RGB to YUV, and the corresponding brightness value of the SAR image is superimposed on the Y channel. The local contrast and information entropy of the infrared image and SAR image are calculated, and the fusion ratio of the infrared image and SAR image is dynamically adjusted to generate a registered fused image.

2. The method for registering and fusing infrared images and SAR images in dynamic flight of a UAV according to claim 1, characterized in that: The measurement and calibration method in step S1 specifically includes: using a reentrant infrared SAR dual-light common field light pipe to gradually measure the Y-direction field of view angle of the infrared camera; the reentrant infrared SAR dual-light common field of view light pipe contains a crosshair, and the step length is 0.1m.

3. The method for registering and fusing infrared images and SAR images in dynamic flight of a UAV according to claim 1, characterized in that: The method for adjusting the field of view angle of a single pixel of the infrared image to match the field of view angle of a single pixel of the SAR image in step S2 is specifically as follows: The onboard computer calculates the focal length of the infrared camera when each pixel of the infrared camera has the same resolution as the ground pixel of the SAR image based on the flight altitude of the UAV and the azimuth and pitch angle of the optoelectronic payload. ; The camera control board uses the PID algorithm to control the infrared camera to adjust to the focal length , until the field of view of a single infrared pixel matches the field of view of a single SAR pixel.

4. The method for registering and fusing infrared images and SAR images in dynamic flight of a UAV according to claim 1, characterized in that: In step S3, the heterogeneous image descriptors of the infrared image and the SAR image include a phase consistency descriptor and a gradient descriptor; the weighted correlation distance method is used for matching; the specific method is as follows: S301. Calculate the phase consistency descriptor of the heterogeneous image, which is used to measure the phase of a specific frequency component in the heterogeneous image; for each pixel, the phase can be calculated by the following formula: ； in, are pixel coordinates, w are different scales, is the phase at scale w, M represents the number of sizes; S302. Calculate the gradient descriptor of the heterogeneous image; The gradient direction descriptor of the heterogeneous image is as follows: ； ； in, and are the gradients in the x and y directions when the scale is m, respectively, and 𝐼 is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as: ； S303. Set the statistical scale to m, and take the (x, y) pixel as the center and the gradient histogram of the surrounding 10×10 pixels as follows: ； S304. Based on the image target size, take M / 2 scales upward and M / 2 scales downward, for a total of M scales; ; Phase consistency descriptor at the mth scale , gradient descriptor , gradient histogram It constitutes the descriptor vector at the mth scale : ； S305. Use the weighted correlation distance method to perform matching; the expression is as follows: ； Where WCD represents the weighted correlation distance, which is the weighted sum of the differences between the infrared image and the SAR image descriptor vectors from scale 1 to scale M; m is the scale of the descriptor; M represents the number of sizes; and Respectively represent the multi-scale descriptor vector of each feature point in the infrared image and the SAR image; is the weight of the mth scale, expressed as: ; is the descriptor vector of the mth scale and The variance of When the WCD value is greater than 0.5, it is considered and The match is successful; the projection relationship between the infrared image and each pixel coordinate in the SAR image is stored as a matrix C.

5. The method for registering and fusing infrared images and SAR images in dynamic flight of a UAV according to claim 1, characterized in that: The step S4 specifically includes the following sub-steps: S401. Calculate the local contrast of the infrared image and the SAR image; normalize the local contrast of the infrared image and the SAR image to obtain the weight of the local contrast of the infrared image and the SAR image; S402. Calculate the information entropy of the infrared image and the information entropy of the SAR image, normalize the information entropy of the infrared image and the SAR image, and obtain the weights of the information entropy of the infrared image and the SAR image; S403. Convert the RGB format file of the infrared image to YUV, and make the Y channel brightness value of the infrared image Y ir and Y channel brightness value of SAR image Y SAR Overlay and fuse pixel by pixel to generate a registered fused image Y fused, the expression is as follows: ； In the formula, Y fused means to register the fused image. Y ir represents the Y channel brightness value of the infrared image, Y SAR is the Y channel brightness value of the SAR image; represents the infrared image weight based on local contrast, represents the SAR image weight based on local contrast; represents the infrared image weight based on information entropy, Represents the SAR image weight based on information entropy.

6. The method for registering and fusing infrared images and SAR images in dynamic flight of a UAV according to claim 5, characterized in that: The step S401 specifically includes the following sub-steps: S4011. Determine a local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel value in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast; The calculation formulas for the local standard deviation of infrared images and SAR images are as follows: ； ； In the formula, is the infrared image at coordinates ( x , y ), is the SAR image at coordinates ( x , y ), is the local area Ω in the infrared image or SAR image f The average pixel value, |Ω f ∣ represents the local area Ω in the infrared image or SAR image f The total number of pixels; The local contrast expression of infrared image or SAR image is as follows: ； ； S4012. Normalize the local contrast of the infrared image and the SAR image so that the sum of IRLocalContrast and SARLocalContrast is 1; obtain the weight of the local contrast of the infrared image and the SAR image: ； In the formula, represents the infrared image weight based on local contrast, Represents the SAR image weight based on local contrast.

7. The method for registering and fusing infrared images and SAR images in dynamic flight of a UAV according to claim 6, characterized in that: The step S402 specifically includes the following sub-steps: S4021. Use a histogram to represent the number of times the gray value 𝑖 appears in the infrared image or SAR image; for each pixel in the gray image, the gray value or , do the following: ； In the formula, and Indicates infrared histogram and SAR histogram; i indicates gray value; represents the indicator function; when When it is equal to i, is 1, otherwise it is 0; When it is equal to i, is 1, otherwise it is 0; Represents the grayscale value of the pixel with coordinates (x, y) in the infrared image. Represents the grayscale value of the pixel with coordinates (x, y) in the SAR image; S4022. Perform normalization to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image: ； ； In the formula, and are the probability distribution of each gray value of infrared image and SAR image respectively, and are the width and height of the infrared image, and are the width and height of the SAR image respectively; S4023. Use the Shannon information entropy formula to multiply the probability of each gray value by its logarithm with base 2, and accumulate all 256 possible gray values ​​to calculate the infrared image information entropy 𝐻ir and the SAR image information entropy 𝐻SAR; the calculation formula is as follows: ； ； S4024. Normalize the information entropy of the infrared image and the SAR image so that H ir and H The sum of SAR is 1, and the weights of the information entropy of the infrared image and the SAR image are obtained: ； In the formula, represents the infrared image weight based on information entropy, Represents the SAR image weight based on information entropy.

8. The method for registering and fusing infrared images and SAR images during dynamic flight of a UAV according to claim 7, characterized in that: The value range of the information entropy is 0 to 8 bits; when all pixels have the same gray value, H ir=0, H SAR=0; when the probability of each gray value appearing is equal, the information entropy H ir=8, H SAR=8.

Images