Visible light and infrared image fusion tracking method in dynamic flight of unmanned aerial vehicle

Abstract

The invention relates to the technical field of unmanned aerial vehicle target tracking, in particular to a visible light and infrared image fusion tracking method in dynamic flight of an unmanned aerial vehicle. Comprising the following steps: measuring and calibrating a relationship between a visible light camera focal length and a visible light Y-direction field angle on the ground, and recording the relationship between the field angle and the focal length as a file; the camera control panel controls the movement of the visible light camera through a PID algorithm until the field angles of single pixels of the visible light image and the infrared image are matched; extracting different-source image descriptors, matching the different-source image descriptors, storing the projection relation of each pixel coordinate as a matrix A, and calling to complete registration; calculating the local contrast and the information entropy of the visible light image and the infrared image, dynamically adjusting the fusion proportion, and generating a registration fusion image; and inputting a related tracking algorithm to track the target. The method has the advantages that simple calibration on the ground is only needed; and after zooming, a registration algorithm is operated on an airborne computer, and the efficiency and accuracy of feature matching are improved in combination with a multi-scale descriptor.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) target tracking, and in particular to a fusion tracking method of visible light and infrared images in the dynamic flight of an unmanned aerial vehicle (UAV). Background Art

[0002] Traditional image registration methods usually require a complex ground calibration process, which includes accurate measurement of the visual axis center offset, edge distortion, and pixel mapping relationship of visible light and infrared cameras. These calibration processes are often time-consuming and labor-intensive, and are only applicable to fixed focal length lenses. However, in the optoelectronic turret system onboard drones, visible light and infrared cameras are usually equipped with zoom lenses, which makes traditional calibration methods no longer applicable because they need to be calibrated separately for each focal length, resulting in a huge workload.

[0003] In addition, due to the dynamic nature of the UAV platform and the real-time requirements of the mission, traditional image registration methods face challenges in practical applications. When UAVs perform monitoring tasks, changes in camera focal length are normal, which requires image registration algorithms to be able to quickly adapt to focal length changes and achieve real-time processing.

[0004] The main limitations of the existing technology are its complex ground calibration process and the lack of image registration solutions suitable for zoom lenses; there is a lack of algorithms that can adapt to the focal length changes of the variable-focal-length visible camera and the variable-focal-length infrared camera on the UAV's airborne optoelectronic turret and achieve fast and accurate registration. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a fusion tracking method of visible light and infrared images in the dynamic flight of a UAV.

[0006] The present invention aims to provide a method for fusion tracking of visible light and infrared images in dynamic flight of unmanned aerial vehicles, which specifically comprises the following steps: S1. Ground calibration: measure and calibrate the relationship between the focal length of the visible light camera and the field angle in the Y direction of the visible light, and the relationship between the focal length of the infrared camera and the field angle in the Y direction of the infrared light, and record the relationship between the field angle and the focal length as a file and store it on the camera control board; S2. Coarse matching of field of view angle: The camera control board controls the movement of the visible light camera or infrared camera through the PID algorithm, so that the field of view angle of a single pixel of the visible light image is equal to the field of view angle of a single pixel of the infrared image; S3. Image descriptor extraction and registration: extract heterogeneous image descriptors of visible light image and infrared image, match the descriptors, store the projection relationship of each pixel coordinate of visible light image and infrared image as matrix A; call matrix A to complete the registration; S4. Image fusion: According to the registered image, convert the visible light image format from RGB to YUV, and superimpose the corresponding brightness value of the infrared image on the Y channel, calculate the local contrast and information entropy of the visible light image and the infrared image, dynamically adjust the fusion ratio of the visible light image and the infrared image, and generate the registered fused image; S5. Input the registered fused image into the relevant tracking algorithm to perform target tracking.

[0007] Preferably, the measurement and calibration method in step S1 specifically includes: using a reentrant visible infrared dual light common field light tube to gradually measure the Y direction field angle of the visible light camera and the Y direction field angle of the infrared camera; the reentrant visible infrared dual light common field light tube contains a crosshair, and the step length is 0.1m.

[0008] Preferably, the method for adjusting the field angle of a single pixel of the visible light image to be equal to the field angle of a single pixel of the infrared image in step S2 is specifically as follows: When the UAV performs a mission, if the visible lens is used as the main lens and infrared is integrated, the following steps are performed: when the visible lens is used as the main lens and the ground station sends a large field of view or small field of view command, only the visible lens responds; after the ground station control ends, the visible lens is in a static state. At this time, the focal length value g of the visible light is read out. CCD ; According to the focal length of visible light g CCD , visible pixel size and infrared pixel size, calculate the infrared focal length of a single pixel under the same field of view, and the camera control board controls the repeated movement of the infrared camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length; When the UAV is performing a mission, if the infrared lens is used as the main lens and the visible lens is integrated, the following steps are performed: when the infrared lens is used as the main lens and the ground station sends a large field of view or a small field of view command, only the infrared lens responds; after the ground station control ends, the infrared lens is in a static state. At this time, the focal length value g of the infrared lens is read out. IR ; According to the focal length value g of the infrared lens IR , according to the visible pixel size and the infrared pixel size, the visible light focal length of a single pixel under the same field of view is calculated. The camera control board controls the repeated movement of the visible light camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length.

[0009] Preferably, in step S3, the heterogeneous image descriptors of the visible light image and the infrared image include a phase consistency descriptor and a gradient descriptor; and the weighted correlation distance method is used for matching; the specific method is as follows: S301. Calculate the phase consistency descriptor of the heterogeneous image, which is used to measure the phase of a specific frequency component in the heterogeneous image; for each pixel, the phase can be calculated by the following formula: ; in, are pixel coordinates, w are different scales, is the phase at scale w, M represents the number of sizes; S302. Calculate the gradient descriptor of the heterogeneous image; The gradient direction descriptor of the heterogeneous image is as follows: ; ; in, and are the gradients in the x and y directions when the scale is m, I is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as:

[0010] ; In the formula, Indicates the direction of the gradient, that is, the direction of the brightness change of the image at the point (x, y); S303. Set the statistical scale to m, and take the pixel (x, y) as the center and the gradient histogram of the surrounding 10×10 pixels as follows: ; S304. Based on the image target size, take M / 2 scales upward and M / 2 scales downward, for a total of M scales; ; Phase consistency descriptor at the mth scale , gradient descriptor , gradient histogram It constitutes the descriptor vector at the mth scale : ; S305. Use the weighted correlation distance method to perform matching; the expression is as follows: ; Where WCD represents the weighted correlation distance, which is the weighted sum of the differences between the descriptor vectors of the visible light image and the infrared image from scale 1 to scale M; m is the scale of the descriptor; M represents the number of sizes; and Represent the multi-scale descriptor vector at the mth scale in the visible light image and infrared image respectively; is the weight of the mth scale, expressed as: ; is the descriptor vector of the mth scale and The variance of When the WCD value is greater than 0.5, it is considered and The match is successful; the projection relationship between the coordinates of each pixel in the visible light image and the infrared image is stored as matrix A.

[0011] Preferably, step S4 specifically includes the following sub-steps: S401. Calculate the local contrast of the visible light image and the infrared image; normalize the local contrast of the visible light image and the infrared image to obtain the weight of the local contrast of the visible light image and the infrared image; S402. Calculate the information entropy of the visible light image and the information entropy of the infrared image, normalize the information entropy of the visible light image and the infrared image, and obtain the weights of the information entropy of the visible light image and the infrared image; S403. Convert the RGB format file of the visible light image to YUV, and make the Y channel brightness value of the visible light image Y CCD and infrared image Y channel brightness value Y IR superposition, pixel-by-pixel fusion, to generate a registered fused image Y fused , the expression is as follows: ; In the formula, Y fused represents the registered fusion image, Y CCD represents the Y channel brightness value of the visible light image. Y ir is the Y channel brightness value of the infrared image; represents the visible light image weight based on local contrast, represents the infrared image weight based on local contrast; represents the weight of the visible light image based on information entropy, Represents the infrared image weight based on information entropy.

[0012] Preferably, step S401 specifically includes the following sub-steps: S4011. Determine a local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel value in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast; The calculation formulas for the local standard deviation of visible light images and infrared images are as follows: ; ; In the formula, is the visible image at coordinates ( x , y ), is the infrared image at coordinates (x , y ), is the local area Ω in the visible light image or infrared image f The average pixel value, |Ω f ∣ represents the local area Ω in the visible light image or infrared image f The total number of pixels; The local contrast expression of a visible light image or infrared image is as follows: CCDLocalContrast=max( ); IRLocalContrast=max( ); S4012. Normalize the local contrast of the visible light image and the infrared image so that the sum of CCDLocalContrast and IRLocalContrast is 1; obtain the weight of the local contrast of the visible light image and the infrared image: ; In the formula, represents the visible light image weight based on local contrast, Represents the infrared image weight based on local contrast.

[0013] Preferably, step S402 specifically includes the following sub-steps: S4021. Use a histogram to represent the number of times the gray value i appears in the visible light image or infrared image; for each pixel in the gray image, the gray value or , do the following: ; In the formula, and Represents visible histogram and infrared histogram; i represents grayscale value; represents the indicator function; when ; when it is equal to i, is 1, otherwise it is 0; When it is equal to i, is 1, otherwise it is 0; Represents the grayscale value of the pixel with coordinates (x, y) in the visible image. Represents the grayscale value of the pixel with coordinates (x, y) in the infrared image; S4022. Perform normalization to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image: ; ; In the formula, and are the probability distribution of each gray value of the visible light image and infrared image respectively, and are the width and height of the visible light image, respectively. and are the width and height of the infrared image respectively; S4023. Using the Shannon information entropy formula, multiply the probability of each gray value by its logarithm with base 2, and accumulate all 256 possible gray values ​​to calculate the information entropy of the visible light image. H CCD and infrared image information entropy H ir; the calculation formula is as follows: ; ; S4024. Normalize the information entropy of the visible light image and the infrared image so that H CCD and H ir is 1, and the weights of the information entropy of the visible light image and the infrared image are obtained: ; In the formula, represents the weight of the visible light image based on information entropy, Represents the infrared image weight based on information entropy.

[0014] Preferably, the value range of information entropy is 0 to 8 bits; when all pixels have the same grayscale value, H CCD = 0, H ir = 0; when the probability of each gray value appearing is equal, the information entropy H CCD = 8, H ir =8.

[0015] Preferably, the correlation tracking algorithm in step S5 specifically includes the following steps: S501. Target region initialization: In the fused tracking image, the initial region of the target is determined by the target detection algorithm, and the initial region is used as the starting point of the tracking algorithm; S502. Template matching and correlation calculation: Use the initial area of ​​the target as a template and search for the target by calculating the correlation between the template and each possible position in the image; the correlation can be calculated by the following formula: ; in, R ( x , y ) is at the position of the registered fusion image ( x ,y ), T ( i , j ) is the pixel value of the fused template image, I ( x + i , y + j ) is the pixel value of the target fused image, μT and μI are the means of the template and target regions, respectively; S503. Peak detection and target positioning: In the correlation graph, the new position of the target is determined by finding the local maximum; if the peak value is higher than the preset threshold, the target is considered to be successfully tracked at this position, and the target model is updated according to the tracking result.

[0016] Compared with the prior art, the present invention can achieve the following beneficial effects: The present invention only needs to simply calibrate the visible and infrared field of view angles on the ground, and after zooming, run the registration algorithm on the onboard computer, combining multi-scale descriptors and optimized search strategies to improve the efficiency and accuracy of feature matching. It is suitable for visible infrared image registration fusion tracking under conditions of different resolutions and visible infrared cameras with zoom lenses, and is particularly suitable for image registration tracking under variable focal length visible cameras and variable focal length infrared cameras on unmanned aerial vehicles. It has broad application prospects and practical application value, significantly reduces the workload of registration and calibration, and improves the reliability of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The present invention is a flowchart of a method for fusing visible light and infrared images in the dynamic flight of a UAV provided in accordance with an embodiment of the present invention.

[0018] Figure 2 It is a visible light image taken by a drone according to an embodiment of the present invention.

[0019] Figure 3 This is an infrared image taken by a drone according to an embodiment of the present invention.

[0020] Figure 4 It is a fused registration image generated by fusing the visible light image and the infrared image on the drone provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0023] The present invention aims to solve the limitations of the prior art and provide a new image registration method, which is applicable to the integrated payload system of the photoelectric radar onboard the UAV, can reduce the calibration workload and improve the efficiency and accuracy of image registration. By performing a simple field of view angle calibration on the ground, combined with multi-scale descriptors and optimized search strategies, the registration algorithm can be run in real time on the onboard computer to meet the needs of real-time image processing of the UAV.

[0024] See also Figure 1 The present invention provides a fusion tracking method of visible light and infrared images in the dynamic flight of a UAV, which specifically includes the following steps: S1. Ground calibration: Install the visible light camera and infrared camera on the optoelectronic turret on the drone; measure and calibrate the relationship between the focal length of the visible light camera and the visible light Y-direction field of view angle and the relationship between the focal length of the infrared camera and the infrared Y-direction field of view angle, record the relationship between the field of view angle and the focal length as a file, and store it on the camera control board; The measurement and calibration method specifically includes: using a reentrant visible infrared dual light common field light tube (the light tube contains crosshairs, and the step length is 0.1m) to gradually measure the Y-direction field angle of the visible light camera and the Y-direction field angle of the infrared camera; Specifically, the optoelectronic turret is equipped with a measured azimuth and pitch encoder; assuming that the focal length of the visible light camera lens is CCD_F at this time, the pitch angle of the optoelectronic turret is locked to 0, align the upper edge of the visible light camera with the crosshairs of the light tube, and read the azimuth angle CCDup of the optoelectronic turret at this time; rotate the optoelectronic turret to align the lower edge of the visible light camera with the crosshairs of the light tube, and read the azimuth angle CCDdown of the optoelectronic turret at this time, then when the focal length of the visible light camera is CCD_F, the visible light Y-direction field of view angle is CCD_Y=CCDup-CCDdown; record the corresponding relationship between the focal length of the visible light lens and the field of view angle in array c; similarly, measure the Y-direction field of view angle of the infrared camera, IR_Y=IRup-IRdown; record the corresponding relationship between the focal length of the infrared lens and the field of view angle in array d, and store it in the optoelectronic turret recorder on the drone.

[0025] S2. Rough matching of field of view angle: The camera control board controls the movement of the visible light camera or infrared camera through the PID algorithm to make the field of view angle of a single pixel of the visible light image equal to the field of view angle of a single pixel of the infrared image. The specific operations are as follows: If each pixel of the visible light image and the infrared image has the same field of view, the relationship between the focal lengths of the visible light lens and the infrared lens is derived as follows: Visible light pixel size: ; Infrared pixel size: ; The focal length of the visible light lens is fvi ; The focal length of the infrared lens is fir ; The angular resolution θ can be expressed by the following formula: θ = pixel size / focal length; The angular resolution of the visible light image is θ vis = dvis / fvis ; The angular resolution of the infrared image is ; To make θ vis Equal to θ ir , the following equation can be established: ; That is, when the visible light pixel size is 2.5 μm and the infrared pixel size is 15 μm, that is, the visible focal length is 1 / 6 of the infrared focal length, the field of view of a single pixel in the visible light image is equal to the field of view of a single pixel in the infrared image.

[0026] The visible light camera and infrared camera use external triggering to ensure that the visible and infrared are exposed at the same time.

[0027] The method of adjusting the field of view angle of a single pixel of a visible light image to be equal to the field of view angle of a single pixel of an infrared image, that is, adjusting the visible focal length to 1 / 6 of the infrared focal length, is as follows: When the UAV performs a mission, if the visible lens is used as the main lens and infrared is integrated, the following steps are performed: when the visible lens is used as the main lens and the ground station sends a large field of view or small field of view command, only the visible lens responds; after the ground station control ends, the visible lens is in a static state. At this time, the focal length value g of the visible light is read out. CCD ; According to the focal length of visible light g CCD , visible pixel size and infrared pixel size, calculate the infrared focal length of a single pixel under the same field of view, and the camera control board controls the repeated movement of the infrared camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length; When the UAV is performing a mission, if the infrared lens is used as the main lens and the visible lens is integrated, the following steps are performed: when the infrared lens is used as the main lens and the ground station sends a large field of view or a small field of view command, only the infrared lens responds; after the ground station control ends, the infrared lens is in a static state. At this time, the focal length value g of the infrared lens is read out. IR ; According to the focal length value g of the infrared lens IR, according to the visible pixel size and the infrared pixel size, the visible light focal length of a single pixel under the same field of view is calculated. The camera control board controls the repeated movement of the visible light camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length.

[0028] S3. Image descriptor extraction and registration: extract heterogeneous image descriptors of visible light image and infrared image, match the descriptors, store the projection relationship of each pixel coordinate of visible light image and infrared image as matrix A; call matrix A to complete the registration; The heterogeneous image descriptors of visible light images and infrared images include phase consistency descriptors and gradient descriptors. The method for matching the descriptors is to extract the common feature descriptors of visible light images and infrared images by combining the phase consistency descriptors and gradient descriptors, and then use the weighted correlation distance (WCD) method for matching. The specific method is as follows: S301. Calculate the phase consistency descriptor of the heterogeneous image, which is used to measure the phase of a specific frequency component in the heterogeneous image; for each pixel, the phase can be calculated by the following formula: ; in, are pixel coordinates, w are different scales, is the phase at scale w, M represents the number of dimensions (usually 10); S302. Calculate the gradient descriptor of the heterogeneous image; The gradient direction descriptor of the heterogeneous image is as follows: ; ; in, and are the gradients in the x and y directions when the scale is m, I is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as:

[0029] ; In the formula, Indicates the direction of the gradient, that is, the direction of the brightness change of the image at the point (x, y); S303. Set the statistical scale to m, and take the pixel (x, y) as the center and the gradient histogram of the surrounding 10×10 pixels as follows: ; S304. Based on the image target size, take M / 2 scales upward and M / 2 scales downward, for a total of M scales. Phase consistency descriptor under , gradient descriptor (gradient magnitude) , gradient histogram It constitutes the descriptor vector at the mth scale : ; The visible light image and infrared image are operated according to the above method, and each feature point in the obtained visible light image and infrared image generates a rich multi-scale descriptor vector and ,This descriptor effectively describes the local structural information of visible and infrared images, and provides a solid foundation for feature matching.

[0030] S305. Use the weighted correlation distance (WCD) method for matching; the expression is as follows: ; Where WCD represents the weighted correlation distance, which is the weighted sum of the differences between the descriptor vectors of the visible light image and the infrared image from scale 1 to scale M; m is the scale of the descriptor (the value of m ranges from 1 to M); M represents the number of sizes (usually 10); and Represent the multi-scale descriptor vector at the mth scale in the visible light image and infrared image respectively; is the weight of the mth scale, expressed as: ; is the descriptor vector of the mth scale and The variance of When the WCD value is greater than 0.5, it is considered and The match is successful; the projection relationship between the coordinates of each pixel in the visible light image and the infrared image is stored as matrix A.

[0031] Principle: After step S2, the visible and infrared fields of view are basically consistent in the Y direction in theory. However, due to the field of view angle calibration error in step S1 and the focus control error in step S2, there are still slight differences in the visible and infrared fields of view, so registration is required. The calculation of the common feature descriptors of visible light images and infrared images is a key step in the matching of visible light images and infrared images. It provides a unique vector representation for each feature point of the visible light image and the infrared image.

[0032] The weighted correlation distance (WCD) method not only considers the Euclidean distance between feature vectors, but also the correlation between feature dimensions. This method is particularly suitable for situations where there are intrinsic connections between feature dimensions.

[0033] Because visible infrared uses external triggering to ensure that visible infrared is exposed at the same time, after exposure, a new frame of visible light image and infrared image arrive at the onboard computer. Only by calling matrix A can the real-time registration of visible infrared be completed, so that the real-time performance of the registration is effectively guaranteed.

[0034] S4. Image fusion: According to the registered image, convert the visible light image format from RGB to YUV, and superimpose the corresponding brightness value of the infrared image on the Y channel, calculate the local contrast and information entropy of the visible light image and the infrared image, dynamically adjust the fusion ratio of the visible light image and the infrared image, and generate a registered fused image; specifically, it includes the following sub-steps: S401. Calculate the local contrast of the visible light image and the infrared image; normalize the local contrast of the visible light image and the infrared image to obtain the weight of the local contrast of the visible light image and the infrared image; specifically include the following sub-steps: S4011. Determine a local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel values ​​in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast.

[0035] Local standard deviation It is a local area The measure of the internal pixel value fluctuation, the local standard deviation calculation formulas for visible light images and infrared images are as follows: ; ; In the formula, is the visible image at coordinates ( x , y ), is the infrared image at coordinates ( x , y ), is the local area Ω in the visible light image or infrared image f The average pixel value, |Ω f ∣ represents the local area Ω in the visible light image or infrared image f The total number of pixels.

[0036] The local standard deviation is calculated by calculating the local region Ω f Inner pixel value and its average value μ fThe sum of the squared deviations of , and then take the square root. This value reflects the dispersion of pixel intensity in the feature area and is an indicator of the complexity of local texture.

[0037] The local contrast of a visible light image or infrared image is defined as the maximum value of its local standard deviation: CCDLocalContrast=max( ); IRLocalContrast=max( ); S4012. Normalize the local contrast of the visible light image and the infrared image so that the sum of CCDLocalContrast and IRLocalContrast is 1; obtain the weight of the local contrast of the visible light image and the infrared image: ; In the formula, represents the visible light image weight based on local contrast, Represents the infrared image weight based on local contrast.

[0038] S402. Calculate the information entropy of the visible light image and the information entropy of the infrared image, normalize the information entropy of the visible light image and the infrared image, and obtain the weights of the information entropy of the visible light image and the infrared image; specifically, the following sub-steps are included: S4021. Use a histogram to represent the number of times the gray value i appears in the visible light image or infrared image; for each pixel in the gray image, the gray value or , do the following: ; In the formula, and Represents visible histogram and infrared histogram; i represents grayscale value; represents the indicator function; when ; when it is equal to i, is 1, otherwise it is 0; When it is equal to i, is 1, otherwise it is 0; Represents the grayscale value of the pixel with coordinates (x, y) in the visible image. Represents the grayscale value of the pixel with coordinates (x, y) in the infrared image; S4022. Perform normalization to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image: ; ; In the formula, and are the probability distribution of each gray value of the visible light image and infrared image respectively, and are the width and height of the visible light image, respectively. and are the width and height of the infrared image respectively.

[0039] S4023. Using the Shannon information entropy formula, multiply the probability of each gray value by its logarithm with base 2, and accumulate all 256 possible gray values ​​to calculate the information entropy of the visible light image. H CCD and infrared image information entropy H ir; ; ; The range of information entropy is from 0 to a maximum of 8 bits. This range reflects the amount of information in an image from complete disorder (the grayscale value of each pixel is random and has equal probability) to complete order (all pixels have the same grayscale value); when all pixels have the same grayscale value (that is, the image is completely uniform), H CCD = 0, H ir=0; when the probability of each gray value appearing is equal, the information entropy H CCD = 8, H ir=8, reaching the maximum value.

[0040] S4024. Normalize the information entropy of the visible light image and the infrared image so that H CCD and H ir is 1, and the weights of the information entropy of the visible light image and the infrared image are obtained: ; In the formula, represents the weight of the visible light image based on information entropy, Represents the infrared image weight based on information entropy.

[0041] S403. Convert the RGB format file of the visible light image to YUV, and make the Y channel brightness value of the visible light image Y CCD and infrared image Y channel brightness value Y IR superposition, pixel-by-pixel fusion, to generate a registered fused image Y fused , the expression is as follows: ; In the formula, Y fusedrepresents the registered fusion image, Y CCD represents the Y channel brightness value of the visible light image. Y ir is the Y channel brightness value of the infrared image; represents the visible light image weight based on local contrast, represents the infrared image weight based on local contrast; represents the weight of the visible light image based on information entropy, Represents the infrared image weight based on information entropy.

[0042] S5. Input the registered fusion image into the relevant tracking algorithm to track the target; the relevant tracking algorithm specifically includes the following steps: S501. Target region initialization: In the fused tracking image, the initial region of the target is determined by the target detection algorithm, and the initial region is used as the starting point of the tracking algorithm; S502. Template matching and correlation calculation: Use the initial area of ​​the target as a template and search for the target by calculating the correlation between the template and each possible position in the image; the correlation can be calculated by the following formula: ; in, R ( x , y ) is at the position of the registered fusion image ( x , y ), T ( i , j ) is the pixel value of the fused template image, I ( x + i , y + j ) is the pixel value of the target fused image, μT and μI are the means of the template and target regions, respectively; S503. Peak detection and target positioning: In the correlation graph, the new position of the target is determined by finding the local maximum; if the peak value is higher than the preset threshold, the target is considered to be successfully tracked at this position, and the target model is updated according to the tracking result.

[0043] In summary, the present invention simplifies the ground calibration process and only needs to calibrate the field of view and imaging parameters of the visible light camera and infrared. After the focal length or parameter changes, the registration algorithm is run on the onboard computer to extract the heterogeneous image descriptors, match the descriptors of the visible light and infrared images, and store the projection relationship of each pixel coordinate of the visible light and infrared as a matrix A. When the focal length or parameter configuration remains unchanged, the matrix A only needs to be calculated once and stored in the onboard computer.

[0044] Because visible light and infrared use synchronous triggering to ensure that visible light and infrared acquire data at the same time, after data acquisition, a new frame of visible light image and infrared image arrive at the onboard computer at the same time, and only need to call matrix A to complete the real-time registration of visible light and infrared, so that the real-time registration is effectively guaranteed. After each frame of image is registered by matrix A, the visible light video format is converted from RGB to YUV, and the corresponding brightness value of infrared is superimposed on the Y channel, and the local contrast and information entropy of visible light and infrared are calculated. According to the local contrast and information entropy, the fusion ratio of visible light Y and infrared brightness values ​​is dynamically adjusted to maximize the information content of the fused image.

[0045] The method of the present invention is suitable for image registration under conditions of different resolutions and when the visible light camera and infrared camera are in variable focal length or variable parameter configuration, and is particularly suitable for the fields of image registration of variable focal length visible light camera and infrared image with integrated load of photoelectric radar onboard unmanned aerial vehicle, etc. It has broad application prospects and practical application value, and significantly reduces the workload of registration and calibration.

[0046] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0047] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A fusion tracking method of visible light and infrared images in dynamic flight of unmanned aerial vehicle, characterized in that: The specific steps include: S1. Ground calibration: measure and calibrate the relationship between the focal length of the visible light camera and the field angle in the Y direction of the visible light, and the relationship between the focal length of the infrared camera and the field angle in the Y direction of the infrared light, and record the relationship between the field angle and the focal length as a file and store it on the camera control board; S2. Coarse matching of field of view angle: The camera control board controls the movement of the visible light camera or infrared camera through the PID algorithm, so that the field of view angle of a single pixel of the visible light image is equal to the field of view angle of a single pixel of the infrared image; S3. Image descriptor extraction and registration: extract heterogeneous image descriptors of visible light image and infrared image, match the descriptors, store the projection relationship of each pixel coordinate of visible light image and infrared image as matrix A; call matrix A to complete the registration; S4. Image fusion: According to the registered image, convert the visible light image format from RGB to YUV, and superimpose the corresponding brightness value of the infrared image on the Y channel, calculate the local contrast and information entropy of the visible light image and the infrared image, dynamically adjust the fusion ratio of the visible light image and the infrared image, and generate the registered fused image; S5. Input the registered fused image into the relevant tracking algorithm to perform target tracking.

2. The method for fusion tracking of visible light and infrared images in dynamic flight of a UAV according to claim 1 is characterized in that: The measurement and calibration method in step S1 specifically includes: using a reentrant visible infrared dual light common field light tube to gradually measure the Y direction field angle of the visible light camera and the Y direction field angle of the infrared camera; the reentrant visible infrared dual light common field light tube contains a crosshair, and the step length is 0.1m.

3. The method for fusion tracking of visible light and infrared images in dynamic flight of a UAV according to claim 2 is characterized in that: The method for adjusting the field of view angle of a single pixel of the visible light image to be equal to the field of view angle of a single pixel of the infrared image in step S2 is specifically as follows: When the UAV performs a mission, if the visible lens is used as the main lens and infrared is integrated, the following steps are performed: when the visible lens is used as the main lens and the ground station sends a large field of view or small field of view command, only the visible lens responds; after the ground station control ends, the visible lens is in a static state. At this time, the focal length value g of the visible light is read out. CCD ; According to the focal length of visible light g CCD , visible pixel size and infrared pixel size, calculate the infrared focal length of a single pixel under the same field of view, and the camera control board controls the repeated movement of the infrared camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length; When the UAV is performing a mission, if the infrared lens is used as the main lens and the visible lens is integrated, the following steps are performed: when the infrared lens is used as the main lens and the ground station sends a large field of view or a small field of view command, only the infrared lens responds; after the ground station control ends, the infrared lens is in a static state. At this time, the focal length value g of the infrared lens is read out. IR ; According to the focal length value g of the infrared lens IR , according to the visible pixel size and the infrared pixel size, the visible light focal length of a single pixel under the same field of view is calculated. The camera control board controls the repeated movement of the visible light camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length.

4. The method for fusion tracking of visible light and infrared images in dynamic flight of a UAV according to claim 1 is characterized in that: In step S3, the heterogeneous image descriptors of the visible light image and the infrared image include a phase consistency descriptor and a gradient descriptor; the weighted correlation distance method is used for matching; the specific method is as follows: S301. Calculate the phase consistency descriptor of the heterogeneous image, which is used to measure the phase of a specific frequency component in the heterogeneous image; for each pixel, the phase can be calculated by the following formula: ； in, are pixel coordinates, w are different scales, is the phase at scale w, M represents the number of sizes; S302. Calculate the gradient descriptor of the heterogeneous image; The gradient direction descriptor of the heterogeneous image is as follows: ； ； in, and They are the gradients in the x and y directions when the scale is m, I is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as: ； In the formula, Indicates the direction of the gradient, that is, the direction of the brightness change of the image at the point (x, y); S303. Set the statistical scale to m, and take the (x, y) pixel as the center and the gradient histogram of the surrounding 10×10 pixels as follows: ； S304. Based on the image target size, take M / 2 scales upward and M / 2 scales downward, for a total of M scales; ; Phase consistency descriptor at the mth scale , gradient descriptor , gradient histogram It constitutes the descriptor vector at the mth scale : ； S305. Use the weighted correlation distance method to perform matching; the expression is as follows: ； Where WCD represents the weighted correlation distance, which is the weighted sum of the differences between the descriptor vectors of the visible light image and the infrared image from scale 1 to scale M; m is the scale of the descriptor; M represents the number of sizes; and Represent the multi-scale descriptor vector at the mth scale in the visible light image and infrared image respectively; is the weight of the mth scale, expressed as: ; is the descriptor vector of the mth scale and The variance of When the WCD value is greater than 0.5, it is considered and The match is successful; the projection relationship between the coordinates of each pixel in the visible light image and the infrared image is stored as matrix A.

5. The method for fusion tracking of visible light and infrared images in dynamic flight of a UAV according to claim 1, characterized in that: The step S4 specifically includes the following sub-steps: S401. Calculate the local contrast of the visible light image and the infrared image; normalize the local contrast of the visible light image and the infrared image to obtain the weight of the local contrast of the visible light image and the infrared image; S402. Calculate the information entropy of the visible light image and the information entropy of the infrared image, normalize the information entropy of the visible light image and the infrared image, and obtain the weights of the information entropy of the visible light image and the infrared image; S403. Convert the RGB format file of the visible light image to YUV, and make the Y channel brightness value of the visible light image Y CCD and infrared image Y channel brightness value Y IR superposition, pixel-by-pixel fusion, to generate a registered fused image Y fused , the expression is as follows: ； In the formula, Y fused represents the registered fusion image, Y CCD represents the Y channel brightness value of the visible light image. Y ir is the Y channel brightness value of the infrared image; represents the visible light image weight based on local contrast, represents the infrared image weight based on local contrast; represents the weight of the visible light image based on information entropy, Represents the infrared image weight based on information entropy.

6. The method for fusion tracking of visible light and infrared images in dynamic flight of a UAV according to claim 5, characterized in that: The step S401 specifically includes the following sub-steps: S4011. Determine a local area centered on the feature, apply a mean filter to the local area to obtain the local average brightness, calculate the standard deviation of the deviation between the pixel value in the local area and the local average brightness, and take the maximum value of the standard deviation as the local contrast; The calculation formulas for the local standard deviation of visible light images and infrared images are as follows: ； ； In the formula, is the visible image at coordinates ( x , y ), is the infrared image at coordinates ( x , y ), is the local area Ω in the visible light image or infrared image f The average pixel value, |Ω f ∣ represents the local area Ω in the visible light image or infrared image f The total number of pixels; The local contrast expression of a visible light image or infrared image is as follows: CCDLocalContrast=max( ); IRLocalContrast=max( ); S4012. Normalize the local contrast of the visible light image and the infrared image so that the sum of CCDLocalContrast and IRLocalContrast is 1; obtain the weight of the local contrast of the visible light image and the infrared image: ； In the formula, represents the visible light image weight based on local contrast, Represents the infrared image weight based on local contrast.

7. The method for fusion tracking of visible light and infrared images in dynamic flight of a UAV according to claim 6, characterized in that: The step S402 specifically includes the following sub-steps: S4021. Use a histogram to represent the number of times the gray value i appears in the visible light image or infrared image; for each pixel in the gray image, the gray value or , do the following: ； In the formula, and Represents visible histogram and infrared histogram; i represents grayscale value; represents the indicator function; when ; when it is equal to i, is 1, otherwise it is 0; When it is equal to i, is 1, otherwise it is 0; Represents the grayscale value of the pixel with coordinates (x, y) in the visible image. Represents the grayscale value of the pixel with coordinates (x, y) in the infrared image; S4022. Perform normalization to convert the histogram into a probability distribution by dividing the frequency of each gray value by the total number of pixels in the image: ； ； In the formula, and are the probability distribution of each gray value of the visible light image and infrared image respectively, and are the width and height of the visible light image, respectively. and are the width and height of the infrared image respectively; S4023. Using the Shannon information entropy formula, multiply the probability of each gray value by its logarithm with base 2, and accumulate all 256 possible gray values ​​to calculate the information entropy of the visible light image. H CCD and infrared image information entropy H ir; the calculation formula is as follows: ； ； S4024. Normalize the information entropy of the visible light image and the infrared image so that H CCD and H ir is 1, and the weights of the information entropy of the visible light image and the infrared image are obtained: ； In the formula, represents the weight of the visible light image based on information entropy, Represents the infrared image weight based on information entropy.

8. The method for fusion tracking of visible light and infrared images in dynamic flight of a UAV according to claim 7, characterized in that: The value range of the information entropy is 0 to 8 bits; when all pixels have the same gray value, H CCD = 0, H ir = 0; when the probability of each gray value appearing is equal, the information entropy H CCD = 8, H ir =8.

9. The method for fusion tracking of visible light and infrared images in dynamic flight of a UAV according to claim 1, characterized in that: The correlation tracking algorithm in step S5 specifically includes the following steps: S501. Target region initialization: In the fused tracking image, the initial region of the target is determined by the target detection algorithm, and the initial region is used as the starting point of the tracking algorithm; S502. Template matching and correlation calculation: Use the initial area of ​​the target as a template and search for the target by calculating the correlation between the template and each possible position in the image; the correlation can be calculated by the following formula: ； in, R ( x , y ) is at the position of the registered fusion image ( x , y ), T ( i , j ) is the pixel value of the fused template image, I ( x + i , y + j ) is the pixel value of the target fused image, μT and μI are the means of the template and target regions, respectively; S503. Peak detection and target positioning: In the correlation graph, the new position of the target is determined by finding the local maximum; if the peak value is higher than the preset threshold, the target is considered to be successfully tracked at this position, and the target model is updated according to the tracking result.

Images