A visible light image enhancement method combining infrared features

Abstract

The present invention relates to the technical field of image registration of unmanned aerial vehicle payload systems, and particularly to a visible light image enhancement method combining infrared features. It includes measuring and calibrating the relationship between the focal length of the visible light camera and the vertical field of view angle of the visible light on the ground, recording the relationship between the field of view angle and the focal length as a file, and storing it on the camera control board; the camera control board controls the movement of the visible light camera through the PID algorithm until the field of view angle of a single pixel of the visible light image matches the field of view angle of a single pixel of the infrared image; extracting and matching the heterogeneous image descriptors, storing the projection relationship of each pixel coordinate as matrix A, and calling to complete the registration; adjusting the areas with insufficient details or low contrast in the visible light image according to the feature information in the infrared image to generate a visible light enhanced image. The advantages are: having the ability of real-time processing, significantly improving the quality of visible light images under different lighting conditions and environmental backgrounds, and enhancing the readability of visible light images and the recognizability of targets.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration of UAV payload systems, and particularly to a visible light image enhancement method combining infrared features. Background Art

[0002] In existing image enhancement technologies, the enhancement of visible light images usually relies on the information carried by the images themselves. However, this method has obvious limitations. Especially in poor lighting conditions or low visibility weather conditions such as heavy fog, it becomes difficult to extract effective information in visible light images, resulting in poor enhancement effects and more noise in the enhanced images.

[0003] Traditional image registration methods usually require a complex ground calibration process, which includes precise measurement of the offset of the optical axis centers of visible light and infrared radars, edge distortion, and pixel mapping relationships. In the prior art, the registration of visible light and infrared images usually requires the execution of a cumbersome ground calibration procedure, including calibrating the offset of the optical axis centers of visible light and infrared, imaging mode differences, pixel mapping relationships, etc. The calibration process is complex and only applicable to the case where both visible light and infrared are fixed focal length lenses. The visible light camera of the optoelectronic radar integrated payload has a variable focal length, and the infrared system has variable parameter configurations. The workload of calibrating the center offset, edge distortion, pixel mapping, etc. for each focal length or parameter configuration is huge. The main limitations of the prior art lie in its complex ground calibration process and the lack of an image registration solution applicable to variable focal length lenses; the lack of an algorithm that can adapt to the changes in the variable focal length visible light camera of the UAV-borne optoelectronic radar integrated payload and the focal length of the variable ground resolution infrared radar and achieve fast and accurate registration. Summary of the Invention

[0004] To solve the above problems, the present invention provides a visible light image enhancement method combining infrared features.

[0005] The purpose of the present invention is to provide a visible light image enhancement method combining infrared features, which specifically includes the following steps:

[0006] S1. Ground calibration: Measure and calibrate the relationship between the focal length of the visible light camera and the vertical field of view angle of visible light and the relationship between the focal length of the infrared camera and the vertical field of view angle of infrared on the ground, record the relationship between the field of view angle and the focal length as a file, and store it on the camera control board;

[0007] S2. Coarse matching of field of view angles: The camera control board controls the movement of the visible light camera or the infrared camera through the PID algorithm to make the field of view angle of a single pixel in the visible light image equal to the field of view angle of a single pixel in the infrared image;

[0008] S3. Image descriptor extraction and registration: Extract heterologous image descriptors of visible light images and infrared images, match the descriptors, and store the projection relationship of each pixel coordinate of the visible light image and the infrared image as matrix A; call matrix A to complete the registration.

[0009] S4. According to the feature information in the infrared image, adjust the areas with insufficient details or low contrast in the visible light image, enhance the edge details of the areas, and generate a visible light enhanced image.

[0010] Preferably, the measurement and calibration method in step S1 specifically includes: using a folding visible-infrared dual-light common field-of-view optical tube to gradually measure the field-of-view angle in the Y direction of the visible light camera and the field-of-view angle in the Y direction of the infrared camera; the folding visible-infrared dual-light common field-of-view optical tube contains a crosshair, and the stepping length is 0.1 m.

[0011] Preferably, the method for adjusting the field-of-view angle of a single pixel of the visible light image to be equal to the field-of-view angle of a single pixel of the infrared image in step S2 is as follows:

[0012] When the unmanned aerial vehicle performs tasks, if it is mainly based on the visible lens and fuses the infrared, the following steps are executed: Taking the visible lens as the main lens, when the ground station sends the large field-of-view and small field-of-view commands, only the visible lens responds; after the ground station control ends, the visible lens is in a static state. At this time, read out the focal length value g of the visible light. CCD ; According to the focal length value g of the visible light CCD , the visible pixel size and the infrared pixel size, calculate the infrared focal length under the condition of the same field-of-view for a single pixel. The camera control board controls the repeated movement of the infrared camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length.

[0013] When the unmanned aerial vehicle performs tasks, if it is mainly based on the infrared lens and fuses the visible lens, the following steps are executed: Taking the infrared lens as the main lens, when the ground station sends the large field-of-view and small field-of-view commands, only the infrared lens responds; after the ground station control ends, the infrared lens is in a static state. At this time, read out the focal length value g of the infrared lens. IR ; According to the focal length value g of the infrared lens IR , according to the visible pixel size and the infrared pixel size, calculate the visible light focal length under the condition of the same field-of-view for a single pixel. The camera control board controls the repeated movement of the visible light camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length.

[0014] Preferably, the heterologous image descriptors of the visible light image and the infrared image in step S3 include phase consistency descriptors and gradient descriptors; the method of weighted correlation distance is used for matching; the specific method is as follows:

[0015] S301. Calculate the phase consistency descriptor of the heterogeneous images, which is used to measure the phase of specific frequency components in the heterogeneous images; for each pixel, the phase can be calculated by the following formula:

[0016] ;

[0017] where, is the pixel coordinate, w is different scales, is the phase at scale w, and M represents the number of sizes;

[0018] S302. Calculate the gradient descriptor of the heterogeneous images;

[0019] The gradient direction descriptor of the heterogeneous images is as follows:

[0020] ; ;

[0021] where, and are the gradients in the x - direction and y - direction at scale m respectively, I is the brightness of the image; thus, the gradient descriptor of the heterogeneous images is expressed as:

[0022]

[0023] ;

[0024] In the formula, represents the direction of the gradient, that is, the direction of the brightness change of the image at the point (x, y);

[0025] S303. Set the statistical scale as m, and the gradient histogram of 10×10 pixels around the pixel (x, y) is as follows:

[0026] ;

[0027] S304. Based on the target size of the image, take M / 2 scales upward and M / 2 scales downward, for a total of M scales; ; the phase consistency descriptor , the gradient descriptor , and the gradient histogram at the m - th scale form the descriptor vector at the m - th scale:

[0028] ;

[0029] S305. Adopt the method of weighted correlation distance for matching; the expression is as follows:

[0030] ;

[0031] Wherein, WCD represents the weighted correlation distance, that is, the weighted sum of the differences between the visible light image and the infrared image descriptor vectors from scale 1 to scale M; m is the scale of the descriptor; M represents the number of scales; and respectively represent the multi-scale descriptor vectors of each feature point in the visible light image and the infrared image; is the weight of the m-th scale, expressed as: ; is the descriptor vector of the m-th scale and of the variance;

[0032] When the WCD value is greater than 0.5, it is considered that and match successfully; the projection relationship of each pixel coordinate in the visible light image and the infrared image is stored as matrix A.

[0033] Preferably, step S4 specifically includes the following sub-steps:

[0034] S401. Use an edge detection operator to identify areas with insufficient details or low contrast in the visible light image, and perform localization through convolution operations to obtain the edge map of the visible light image;

[0035] S402. Identify areas in the edge map of the visible light image where the edge intensity is lower than the threshold T ; Apply the region growing algorithm to expand the identified areas with weak edges, so that the areas include all connected pixels with weak edges, and the weak edge areas of the visible light image are obtained after processing;

[0036] S403. Adjust the weak edge areas in the visible light image according to the feature information in the infrared image to obtain an enhanced visible light image.

[0037] Preferably, the localization formula in step S401 is:

[0038] ;

[0039] Among them, Edges is the edge map of the visible light image, represents the convolution operation, G represents the edge detection operator, I ccd is the visible light image;

[0040] The edge detection operator G is expressed as:

[0041] .

[0042] Preferably, the weak edge region expression is as follows:

[0043] ;

[0044] where WeakEdges represents the weak edge region, Edges represents the edge map of the visible light image, and T represents the threshold value.

[0045] Preferably, step S403 specifically includes the following sub-steps:

[0046] S4031. Enhance the contrast of the infrared image by normalizing the infrared image features. The infrared image after enhanced contrast is , and the expression is as follows:

[0047] ;

[0048] S4032. Use the infrared image after enhanced contrast to adjust the pixel values in the visible light image to obtain the enhanced visible light image. The adjustment is carried out through the following formula:

[0049] ;

[0050] In the formula, is the enhanced visible light image, represents the visible light image, k is the enhancement coefficient, and 0 ≤ k ≤ 1.

[0051] Preferably, the enhancement coefficient k is determined by the clamp function. The definition of the clamp function is as follows:

[0052] ;

[0053] In the formula, x represents the value of k, min_val is set to 0, and max_val is set to 1;

[0054] To ensure that k does not exceed the range of 0 to 1, the original value of k is set as follows:

[0055] ;

[0056] where, and are the standard deviations of the infrared image after enhanced contrast and the enhanced visible light image respectively. If the original k value is less than 0, the clamp function sets k to 0; if the original k value is greater than 1, k is set to 1; otherwise, the original k value is used.

[0057] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0058] 1. Improve image quality: By combining infrared features, the present invention can effectively improve the quality of visible light images under low visibility conditions, especially under harsh weather conditions such as heavy fog. The enhanced image has clearer details and higher contrast.

[0059] 2. Enhance real-time performance: The simplified ground calibration process and efficient airborne registration algorithm enable the present invention to achieve real-time image processing, meeting the requirements of the UAV dynamic monitoring platform for rapid response.

[0060] 3. Reduce calibration workload: Avoid the need for complex calibration for each focal length in traditional technologies. Only the field of view angle needs to be calibrated, greatly reducing the calibration workload and implementation costs.

[0061] 4. Adapt to varifocal lenses: It is particularly suitable for variable focal length cameras used in UAV airborne optoelectronic turrets, solving the problem that it is difficult to achieve effective registration for varifocal lenses in existing technologies.

[0062] 5. Enhance target recognition ability: By fusing infrared image features, the recognizability of targets in visible light images is enhanced, which is of great significance for application scenarios such as military reconnaissance, traffic monitoring, and environmental monitoring.

[0063] 6. Optimize computational efficiency: Through the optimized algorithm running on the airborne computer, efficient image processing can be achieved with limited computational resources, suitable for use on mobile platforms such as UAVs.

[0064] 7. Improve operation convenience: The simplified calibration process and automated image enhancement process make the present invention easy to operate and integrate, improving the convenience of users using UAVs for image acquisition and analysis.

[0065] 8. Enhance environmental adaptability: Since the present invention can work stably under different lighting and environmental conditions, the reliability and effectiveness of UAVs in field operations and complex environments are improved.

[0066] In summary, the present invention is particularly suitable for dynamic monitoring platforms such as UAVs, and has important practical application value and market potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a flowchart of a visible light image enhancement method combining infrared features according to an embodiment of the present invention.

[0068] Figure 2 is a visible light image of an aerial photograph taken by a UAV according to an embodiment of the present invention.

[0069] Figure 3 is an infrared image of an aerial photograph taken by a UAV according to an embodiment of the present invention.

[0070] Figure 4 is an enhanced visible light image combined with infrared features on the drone provided according to an embodiment of the present invention. Detailed implementation manners

[0071] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are denoted by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, their detailed descriptions will not be repeated.

[0072] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention.

[0073] The present invention aims to solve the limitations of the prior art and provides a new image registration method, which is applicable to the integrated payload system of the airborne optoelectronic radar of the drone, can reduce the calibration workload, and improve the efficiency and accuracy of image registration. By simply calibrating the field of view angle on the ground and combining multi-scale descriptors and an optimized search strategy, the registration algorithm can run in real time on the airborne computer to meet the requirements of real-time image processing of the drone.

[0074] See Figure 1 , the present invention provides a method for enhancing a visible light image combined with infrared features, specifically including the following steps:

[0075] S1. Ground calibration: Install the visible light camera and the infrared camera on the airborne optoelectronic turret of the drone; Measure and calibrate the relationship between the focal length of the visible light camera and the field of view angle in the Y direction of the visible light, and the relationship between the focal length of the infrared camera and the field of view angle in the Y direction of the infrared camera on the ground, record the relationship between the field of view angle and the focal length as a file, and store it on the camera control board;

[0076] The measurement and calibration method specifically includes: Using a folding visible and infrared double-light co-field-of-view light pipe (the light pipe contains a crosshair, and the step length is 0.1 m) to gradually measure the field of view angle in the Y direction of the visible light camera and the field of view angle in the Y direction of the infrared camera;

[0077] Specifically, the optoelectronic turret is internally equipped with measured azimuth and elevation encoders. Assume that the focal length of the visible light camera lens is CCD_F at this time, the elevation angle of the optoelectronic turret is locked at 0, align the upper edge of the visible light camera with the crosshair of the light pipe, and read the azimuth angle CCDup of the optoelectronic turret at this time. Rotate the optoelectronic turret so that the lower edge of the visible light camera aligns with the crosshair of the light pipe, and read the azimuth angle CCDdown of the optoelectronic turret at this time. Then, when the focal length of the visible light camera is CCD_F, the field of view angle in the Y direction of the visible light is CCD_Y = CCDup - CCDdown; record the corresponding relationship between the focal length of the visible light lens and the field of view angle into the array c. Similarly, measure the field of view angle in the Y direction of the infrared camera, IR_Y = IRup - IRdown; record the corresponding relationship between the focal length of the infrared lens and the field of view angle into the array d, and store it in the optoelectronic turret recorder on the UAV.

[0078] S2. Coarse matching of field of view angle: The camera control board controls the movement of the visible light camera or the infrared camera through the PID algorithm to make the field of view angle of a single pixel in the visible light image equal to the field of view angle of a single pixel in the infrared image; the specific operation is as follows:

[0079] If each pixel in the visible light image and the infrared image is to have the same field of view angle, the derivation of the focal length relationship between the visible light lens and the infrared lens is as follows:

[0080] Visible light pixel size: ; Infrared pixel size: ; The focal length of the visible light lens is fvi ; The focal length of the infrared lens is fir ;

[0081] The angular resolution θ can be expressed by the following formula: θ = pixel size / focal length;

[0082] The angular resolution of the visible light image is θ vis = dvis / fvis ; The angular resolution of the infrared image is ;

[0083] To make θ vis equal to θ ir , the following equation can be established: ; ; That is, in the case where the visible light pixel size is 2.5 μm and the infrared pixel size is 15 μm, that is, when the visible light focal length is 1 / 6 of the infrared focal length, the field of view angle of a single pixel in the visible light image is equal to the field of view angle of a single pixel in the infrared image.

[0084] The visible light camera and the infrared camera adopt the external trigger method to ensure that the visible light and the infrared are exposed at the same time.

[0085] Adjust the field of view angle of a single pixel in the visible light image to be equal to that of a single pixel in the infrared image, that is, the method of adjusting the visible focal length to be 1 / 6 of the infrared focal length is as follows:

[0086] When the UAV performs a mission, if the visible lens is the main one and the infrared is fused, the following steps are executed: Taking the visible lens as the main lens, when the ground station sends the large field of view and small field of view commands, only the visible lens responds; after the ground station control ends, the visible lens is in a static state. At this time, read out the focal length value g of the visible light CCD ; According to the focal length value g of the visible light CCD , the visible pixel size and the infrared pixel size, calculate the infrared focal length under the condition of the same field of view for a single pixel. The camera control board controls the repeated movement of the infrared camera through the PID algorithm until the visible focal length is 1 / 6 of the infrared focal length;

[0087] When the UAV performs a mission, if the infrared lens is the main one and the visible lens is fused, the following steps are executed: Taking the infrared lens as the main lens, when the ground station sends the large field of view and small field of view commands, only the infrared lens responds; after the ground station control ends, the infrared lens is in a static state. At this time, read out the focal length value g of the infrared lens IR ; According to the focal length value g of the infrared lens IR , according to the visible pixel size and the infrared pixel size, calculate the visible light focal length under the condition of the same field of view for a single pixel. The camera control board controls the repeated movement of the visible light camera through the PID algorithm until the visible focal length is 1 / 6 of the infrared focal length.

[0088] S3. Image descriptor extraction and registration: Extract the heterogeneous image descriptors of the visible light image and the infrared image, match the descriptors, and store the projection relationship of each pixel coordinate of the visible light image and the infrared image as matrix A; Call matrix A to complete the registration;

[0089] The heterogeneous image descriptors of the visible light image and the infrared image include phase consistency descriptors and gradient descriptors; The method of matching the descriptors is to combine the phase consistency descriptors and the gradient descriptors, extract the common feature descriptors of the visible light image and the infrared image, and use the method of weighted correlation distance (WCD) for matching; The specific method is as follows:

[0090] S301. Calculate the phase consistency descriptor of the heterogeneous image to measure the phase of specific frequency components in the heterogeneous image; For each pixel, the phase can be calculated by the following formula:

[0091] ;

[0092] Among them, is the pixel coordinate, w is different scales, is the phase at scale w, and M represents the number of scales (usually taking the value of 10);

[0093] S302. Calculate the gradient descriptors of the heterogeneous images;

[0094] The gradient direction descriptors of the heterogeneous images are as follows:

[0095] ; ;

[0096] where and are the gradients in the x - direction and y - direction at scale m respectively, I is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as:

[0097]

[0098] ;

[0099] In the formula, represents the direction of the gradient, that is, the direction of the brightness change of the image at the point (x, y);

[0100] S303. Set the statistical scale as m, and the gradient histogram of 10×10 pixels around the center of the (x, y) pixel is expressed as follows:

[0101] ;

[0102] S304. Based on the target size of the image, take M / 2 scales upward and M / 2 scales downward, a total of M scales. The phase consistency descriptor at the m - th scale , the gradient descriptor (gradient amplitude) , and the gradient histogram constitute the descriptor vector at the m - th scale:

[0103] ;

[0104] Both the visible - light image and the infrared image are operated according to the above method, and each feature point in the obtained visible - light image and infrared image generates a rich multi - scale descriptor vector and , which effectively describes the local structure information of the visible image and the infrared image and provides a solid foundation for feature matching.

[0105] S305. Perform matching using the method of Weighted Correlation Distance (WCD); the expression is as follows:

[0106] ;

[0107] In the formula, WCD represents the weighted correlation distance, that is, the weighted sum of the differences between the visible light image and the infrared image descriptor vectors from scale 1 to scale M; m is the scale of the descriptor (the value range of m is from 1 to M); M represents the number of sizes (usually taking the value of 10); and respectively represent the multi-scale descriptor vectors of each feature point in the visible light image and the infrared image; is the weight of the m-th scale, expressed as: ; is the descriptor vector of the m-th scale and of the variance;

[0108] When the WCD value is greater than 0.5, it is considered that and the matching is successful; store the projection relationship of each pixel coordinate in the visible light image and the infrared image as matrix A.

[0109] Principle brief: After step S2, it is theoretically ensured that the visible and infrared fields of view are basically the same in the Y direction. However, due to the field of view angle calibration error in step S1 and the focal length control error in step S2, there are still slight differences in the visible and infrared fields of view. Therefore, registration operations are required. The calculation of the common feature descriptors of the visible light image and the infrared image is the key step in the matching of the visible light image and the infrared image, which provides a unique vector representation for each feature point of the visible light image and the infrared image.

[0110] The method of using weighted correlation distance (WCD) not only considers the Euclidean distance between feature vectors but also considers the correlation between feature dimensions. This method is particularly suitable for situations where there are internal connections between feature dimensions.

[0111] Because the visible and infrared use the external trigger method to ensure that the visible and infrared are exposed at the same moment. After exposure, a new frame of visible light image and infrared image arrives at the airborne computer. Just call matrix A to complete the real-time registration of the visible and infrared, effectively ensuring the real-time performance of the registration.

[0112] Through a simplified ground calibration process, it is only necessary to calibrate the field of view angles and imaging parameters of the visible light camera and the infrared camera. After zooming or parameter changes, run the registration algorithm on the on-board computer, extract the heterogeneous image descriptors, match the descriptors of the visible light and infrared images, and store the projection relationship of each pixel coordinate of the visible light and infrared as matrix A. When the focal length or parameter configuration remains unchanged, matrix A only needs to be calculated once and stored in the on-board computer. Since the visible light and infrared use a synchronous triggering method to ensure that the visible light and infrared obtain data at the same time, after the data is obtained, a new frame of visible light image and infrared image arrive at the on-board computer simultaneously. Just by calling matrix A, the real-time registration of the visible light and infrared can be completed, effectively guaranteeing the real-time performance of the registration.

[0113] S4. According to the feature information in the infrared image, adjust the areas with insufficient details or low contrast in the visible light image, enhance the edge details of the areas, and generate a visible light enhanced image; specifically, it includes the following sub-steps:

[0114] S401. Use an edge detection operator to identify the areas with insufficient details or low contrast in the visible light image, perform localization through convolution operations, and obtain the edge map of the visible light image;

[0115] The localization formula is:

[0116] ;

[0117] Among them, Edges is the edge map of the visible light image, represents the convolution operation, G represents the edge detection operator, I ccd is the visible light image;

[0118] The edge detection operator G can be expressed as:

[0119] ;

[0120] In a specific embodiment, the edge detection operator is a Gaussian difference operator.

[0121] S402. Identify the areas in the edge map of the visible light image where the edge intensity is lower than the threshold T ; Apply the region growing algorithm to expand the identified areas with weak edges, so that the areas include all connected pixels with weak edges, and obtain the weak edge region of the visible light image after processing;

[0122] The expression of the weak edge region is as follows:

[0123] ;

[0124] Among them, WeakEdges represents the weak edge region, Edges represents the edge map of the visible light image, and T represents the threshold;

[0125] S403. Adjust the weak edge region in the visible light image I ir according to the feature information in ccd to obtain an enhanced visible light image; specifically, it includes the following sub-steps:

[0126] S4031. Enhance the contrast of the infrared image by normalizing the infrared image features. The infrared image after enhanced contrast is , and the expression is as follows:

[0127] ;

[0128] S4032. Use the infrared image after enhanced contrast to adjust the pixel values in the visible light image to obtain an enhanced visible light image; the adjustment is carried out through the following formula:

[0129] ;

[0130] In the formula, is the enhanced visible light image, represents the visible light image, and k is the enhancement coefficient used to control the influence degree of infrared features during the enhancement process;

[0131] The enhancement coefficient k is determined through the clamp function; the definition of the clamp function is as follows:

[0132] ;

[0133] In the formula, x represents the value of k, min_val is set to 0, and max_val is set to 1;

[0134] To ensure that k does not exceed the range of 0 to 1, the original value of k is set as follows:

[0135] ;

[0136] Among them, and are the standard deviations of the infrared image after enhanced contrast and the enhanced visible light image respectively, and are used to calculate the original value of k; if the original k value is less than 0, the clamp function sets k to 0; if the original k value is greater than 1, then k is set to 1; otherwise, the original k value is used.

[0137] Brief principle: The value of the enhancement coefficient k can be based on and the infrared image Iir The local statistical characteristics are dynamically determined to ensure that the enhancement effect is natural and not excessive; the value of the enhancement coefficient k is crucial for controlling the degree of image enhancement; the clamp function is used to limit the value of k within a preset range to ensure that the enhancement effect is neither insufficient nor excessive.

[0138] Through the above steps, the present invention can effectively identify and enhance details in visible light images, while using the information provided by infrared images to improve the image quality, especially under low visibility conditions.

[0139] It should be understood that various forms of the processes shown above can be used, steps can be reordered, added or deleted. For example, the steps described in the disclosure of the present invention can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and no limitation is imposed herein.

[0140] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub - combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A visible light image enhancement method combining infrared features, characterized in that: The specific steps include: S1. Ground calibration: measure and calibrate the relationship between the focal length of the visible light camera and the field angle in the Y direction of the visible light, and the relationship between the focal length of the infrared camera and the field angle in the Y direction of the infrared light, and record the relationship between the field angle and the focal length as a file and store it on the camera control board; S2. Coarse matching of field of view angle: The camera control board controls the movement of the visible light camera or infrared camera through the PID algorithm, so that the field of view angle of a single pixel of the visible light image is equal to the field of view angle of a single pixel of the infrared image; S3. Image descriptor extraction and registration: extract heterogeneous image descriptors of visible light image and infrared image, match the descriptors, store the projection relationship of each pixel coordinate of visible light image and infrared image as matrix A; call matrix A to complete the registration; S4. According to the characteristic information in the infrared image, adjust the area with insufficient details or low contrast in the visible light image, enhance the edge details of the area, and generate a visible light enhanced image; the adjustment method specifically includes: The contrast of the infrared image is enhanced by normalizing the infrared image features. The infrared image after contrast enhancement is: , the expression is as follows: ； Using contrast-enhanced infrared images The pixel values ​​in the visible light image are adjusted to obtain an enhanced visible light image; the adjustment is performed using the following formula: ； In the formula, is the enhanced visible light image, Represents a visible light image, k is the enhancement coefficient, 0≤k≤1.

2. The visible light image enhancement method combined with infrared features according to claim 1, characterized in that: The measurement and calibration method in step S1 specifically includes: using a reentrant visible infrared dual light common field light tube to gradually measure the Y direction field angle of the visible light camera and the Y direction field angle of the infrared camera; the reentrant visible infrared dual light common field light tube contains a crosshair, and the step length is 0.1m.

3. The visible light image enhancement method combined with infrared features according to claim 2, characterized in that: The method for adjusting the field of view angle of a single pixel of the visible light image to be equal to the field of view angle of a single pixel of the infrared image in step S2 is specifically as follows: When the UAV performs a mission, if the visible lens is used as the main lens and infrared is integrated, the following steps are performed: when the visible lens is used as the main lens and the ground station sends a large field of view or small field of view command, only the visible lens responds; after the ground station control ends, the visible lens is in a static state. At this time, the focal length value g of the visible light is read out. CCD ; According to the focal length of visible light g CCD , visible pixel size and infrared pixel size, calculate the infrared focal length of a single pixel under the same field of view, and the camera control board controls the repeated movement of the infrared camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length; When the UAV is performing a mission, if the infrared lens is used as the main lens and the visible lens is integrated, the following steps are performed: when the infrared lens is used as the main lens and the ground station sends a large field of view or a small field of view command, only the infrared lens responds; after the ground station control ends, the infrared lens is in a static state. At this time, the focal length value g of the infrared lens is read out. IR ; According to the focal length value g of the infrared lens IR , according to the visible pixel size and the infrared pixel size, the visible light focal length of a single pixel under the same field of view is calculated. The camera control board controls the repeated movement of the visible light camera through the PID algorithm until the visible light focal length is 1 / 6 of the infrared focal length.

4. The visible light image enhancement method combined with infrared features according to claim 1, characterized in that: In step S3, the heterogeneous image descriptors of the visible light image and the infrared image include a phase consistency descriptor and a gradient descriptor; the weighted correlation distance method is used for matching; the specific method is as follows: S301. Calculate the phase consistency descriptor of the heterogeneous image, which is used to measure the phase of a specific frequency component in the heterogeneous image; for each pixel, the phase can be calculated by the following formula: ； in, are pixel coordinates, w are different scales, is the phase at scale w, M represents the number of sizes; S302. Calculate the gradient descriptor of the heterogeneous image; The gradient direction descriptor of the heterogeneous image is as follows: ； ； in, and are the gradients in the x-direction and y-direction when the scale is m, and 𝐼 is the brightness of the image; thus, the gradient descriptor of the heterogeneous image is expressed as: ； S303. Set the statistical scale to m, and take the pixel (x, y) as the center and the gradient histogram of the surrounding 10×10 pixels as follows: ； S304. Based on the image target size, take M / 2 scales upward and M / 2 scales downward, for a total of M scales; ; Phase consistency descriptor at the mth scale , gradient descriptor , gradient histogram It constitutes the descriptor vector at the mth scale : ； S305. Use the weighted correlation distance method to perform matching; the expression is as follows: ； Where WCD represents the weighted correlation distance, which is the weighted sum of the differences between the descriptor vectors of the visible light image and the infrared image from scale 1 to scale M; m is the scale of the descriptor; M represents the number of sizes; and A multi-scale descriptor vector representing each feature point in the visible light image and the infrared image respectively; is the weight of the mth scale, expressed as: ; is the descriptor vector of the mth scale and The variance of When the WCD value is greater than 0.5, it is considered and The match is successful; the projection relationship between the coordinates of each pixel in the visible light image and the infrared image is stored as matrix A.

5. The visible light image enhancement method combined with infrared features according to claim 1, characterized in that: The step S4 specifically includes the following sub-steps: S401. Using an edge detection operator to identify areas with insufficient details or low contrast in the visible light image, positioning them through a convolution operation, and obtaining an edge map of the visible light image; S402. Identify in the edge map of the visible light image that the edge strength is lower than the threshold T region; applying the region growing algorithm to expand the identified weak edge region so that the region includes all connected weak edge pixels, and after processing, the weak edge region of the visible light image is obtained; S403. According to the feature information in the infrared image, the weak edge region in the visible light image is adjusted to obtain an enhanced visible light image.

6. The visible light image enhancement method combined with infrared features according to claim 5, characterized in that: The positioning formula in step S401 is: ； Among them, Edges is the edge map of the visible light image, represents the convolution operation, G represents the edge detection operator, I ccd is a visible light image; The edge detection operator G is expressed as: 。 7. The visible light image enhancement method combined with infrared features according to claim 6, characterized in that: The weak edge region expression is as follows: ； Among them, WeakEdges represents the weak edge area, Edges represents the edge map of the visible light image, and T represents the threshold.

8. The visible light image enhancement method combined with infrared features according to claim 7, characterized in that: The enhancement coefficient k is determined by the clamp function; the definition of the clamp function is as follows: ； In the formula, x represents the value of k, min_val is set to 0, and max_val is set to 1; To ensure that k does not exceed the range of 0 to 1, set the original value of k as follows: ； in, and They are infrared images after contrast enhancement. And enhanced visible light image The clamp function sets k to 0 if the original k value is less than 0, sets k to 1 if the original k value is greater than 1, and uses the original k value otherwise.

Images