Automatic calculation method of MTF of infrared system based on real-life images

Through automatic target area extraction algorithm and improved edge method, the real-time and accuracy of infrared system performance evaluation is solved, efficient and accurate MTF calculations are realized in real-time images, and real-time performance evaluation and health of infrared systems are supported. manage.

CN114022795BActive Publication Date: 2024-12-03BEIHANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111351668.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-16
Publication Date
2024-12-03
Estimated Expiration
2041-11-16

AI Technical Summary

Technical Problem

The existing infrared system performance evaluation methods are mainly carried out in the laboratory, the operation is complex and not real-time, and the real-time performance of the onboard infrared system cannot be reflected in time. Manual selection of the target area can easily lead to inaccurate calculation results.

Method used

Using automatic target area extraction algorithm and improved edge edge method, the modulation transfer function (MTF) of infrared system is calculated through image edge edge detection and target area search, including sliding window, gray value variance filtering, and root mean square fitting error and Fermi function processing improves the accuracy and robustness of the calculation.

Benefits of technology

It realizes the automatic calculation of the MTF of the infrared system in real-time images, breaks away from the constraints of the laboratory, saves manpower and time costs, improves the accuracy of target area selection, enhances the robustness of image noise, and supports infrared system Real-time performance evaluation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114022795B_ABST
    Figure CN114022795B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for automatically calculating the MTF of an infrared system based on real-shot images, which is used for real-time performance evaluation of infrared systems. The MTF automatic calculation method is mainly divided into three parts: 1) image edge detection, which selects the edge straight line edge in the image through image preprocessing and a suitable algorithm; 2) automatic target area search and extraction, the algorithm uses two search methods to automatically extract the target area that meets specific conditions in the infrared real-shot image; 3) calculation of the modulation transfer function MTF, which uses an improved edge method to estimate the MTF of the infrared system through the target area. The main innovation of the invention is that it can automatically and accurately extract the target area required by the edge method from the infrared image, and make certain improvements to the intermediate variable processing process of the edge method. Compared with the method of manually extracting the area to estimate the MTF, the algorithm greatly improves the rapidity of the MTF calculation and the accuracy of the numerical value, suppresses the high-frequency noise caused by sub-pixel sampling, and has important significance for the real-time and accurate estimation of the performance of the infrared system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of infrared system performance evaluation and digital image processing, and in particular relates to an infrared system MTF automatic calculation method based on real-shot images. Background Art

[0002] Airborne infrared systems are detection and reconnaissance devices widely equipped on advanced combat aircraft. They are mainly used for passive target search, precise tracking and weapon guidance. They are the main weapon against the new generation of radar stealth aircraft. Due to the importance of their role, infrared systems have an important strategic position in aircraft weapons and equipment, and their quality is directly related to the combat performance of fighters.

[0003] Detection and guidance weapon devices with infrared systems as the core have long been widely used in my country's third-generation and fourth-generation fighters. The number of equipment is huge, and the quality and performance are also uneven. If the operating status of the system cannot be evaluated in time, and only warnings are given when its performance is very poor, it will greatly affect the progress of the fighter mission and bring strong uncertainty to the combat mission. In addition, severely degraded infrared systems are likely to be damaged during missions, which will increase subsequent maintenance costs. Therefore, timely and accurate performance evaluation of in-service infrared systems, and then life prediction and health management are important ways to ensure the reliable operation of in-service systems.

[0004] Modulation Transfer Function (MTF) is a typical indicator for evaluating the performance of infrared systems, and its size directly affects the quality of infrared system imaging. At present, the method of calculating MTF for infrared system performance evaluation is almost always completed in the laboratory, using the target plate imaging and the knife edge method calculation specified in ISO12233. Although this method is accurate, it has problems such as complex experimental environment layout and cumbersome experimental steps. Moreover, it is unrealistic for infrared systems to perform laboratory performance testing after each mission. Therefore, it is very important to perform performance testing based on real images.

[0005] In the process of obtaining MTF using real-life images, a common method is to manually select effective target areas and calculate them using the knife-edge method. This process consumes a lot of time and energy for researchers, and the extracted target area is only a standard area intuitively perceived by the human eye, and is not necessarily the best area in the entire image that can reflect the system MTF. If the target area is selected incorrectly, the calculated result may be far from the true value. In response to the above problems, the present invention proposes a new algorithm that can automatically retrieve the best target area in an image, and improves the knife-edge method, which can help researchers efficiently obtain the best MTF curve in infrared images for real-time performance evaluation of the system. Summary of the invention

[0006] The present invention proposes a novel method for automatically estimating the MTF of an infrared system by using real-shot images, which is used for real-time evaluation of the performance of an airborne infrared system.

[0007] To achieve the above object, the present invention adopts the following scheme:

[0008] Firstly, the real-shot image is processed by the automatic target region extraction algorithm, which mainly includes two parts: image edge detection and target region search. In the image edge detection module, the algorithm performs preprocessing such as filtering and cropping on the real-shot image; the line segment detector (LSD) algorithm is used to obtain the straight line segments in the image; the line segments are screened using the length, angle and grayscale value conditions on both sides, and the line segment group that meets the above standards is called the edge set. In the target region search module, the algorithm sequentially takes the edge segments in the edge set, constructs a sliding window of a certain size, and scans from one vertex of the edge to another vertex with a certain step size; calculates the grayscale value variance of the area circled by the sliding window; performs secondary selection on the area within the variance threshold, calculates the root mean square fitting error and grayscale value variance of the edge in the area, and adds the two with a certain weight as the measurement standard for secondary selection; sorts the areas according to the measurement standard, and selects the top ten as candidate areas.

[0009] Secondly, the modulation transfer function calculation module is used to obtain the MTF of the infrared system. The specific steps are as follows: use the least squares method to fit the edge expression, the pixel projection of the candidate area, and 4 times oversampling to obtain the ESF; improve the intermediate variable processing process of the edge method: perform data preprocessing and limiting filtering on the ESF, and limit it with the standard of the Fermi function to remove the areas where the root mean square error on both sides of the data point is greater than the set threshold; perform derivatives, Fourier transforms, normalization and other operations on the ESF of the remaining areas to obtain the MTF, and remove data values ​​with excessive low-frequency noise based on experience; according to the order of the candidate areas in the previous step, the MTF calculated for the first area in the candidate area set is used as the estimated value of the actual MTF of the infrared system.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] (1) It is no longer limited to measuring the MTF of infrared systems in laboratory conditions, but can estimate the real-time performance indicators of infrared systems during operation through real-life images.

[0012] (2) A new automatic target area extraction algorithm was designed. Compared with manually obtaining the calculation area of ​​the blade edge method from the image, this method not only saves a lot of manpower and time costs, but also greatly improves the accuracy of target area selection, thus avoiding the situation where the infrared system performance estimation is incorrect due to improper area selection.

[0013] (3) In the calculation process of the blade edge method, the limiting filtering, standard function fitting, and the method of removing non-standard points based on experience are used to limit the intermediate variables ESF and MTF, exclude areas with excessive low-frequency noise from the results, and enhance the robustness of the algorithm to image noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings of the description are used to provide an understanding of the present invention, and together with the embodiments, explain the present invention in detail.

[0015] Figure 1 The figure is a flow chart of the automatic calculation method of MTF of an infrared system based on real-shot images according to the present invention.

[0016] Figure 2 Schematic diagram of the flow of the modulation transfer function calculation module.

[0017] Figure 3 Schematic diagram of the process of automatic target region extraction algorithm.

[0018] Figure 4 Schematic diagram of gray value condition determination in the image edge detection module.

[0019] Figure 5 Schematic diagram of the effect of the line segment detector (LSD) algorithm.

[0020] Figure 6 Schematic diagram of the sliding window construction in the target area search module.

[0021] Figure 7 Schematic diagram of the edge area when calculating the regional gray value variance.

[0022] Figure 8 Schematic diagram of the calculation process of the knife edge method in the international standard ISO12233.

[0023] Figures 9-10 They are respectively the infrared real-shot image of Example 1 and the target area automatically extracted by the algorithm of the present invention, the ESF and MTF values ​​of the target area and the fitting curve.

[0024] Figures 11-12 They are respectively the infrared real-shot image of Example 2 and the target area automatically extracted by the algorithm of the present invention, the ESF and MTF values ​​of the target area and the fitting curve. DETAILED DESCRIPTION

[0025] The specific implementation steps of the above method are described in detail below with reference to the accompanying drawings and examples. It should be noted that the embodiments described in the specification are only a part of the embodiments, not all of the embodiments, and the following embodiments or drawings are only used to illustrate the present invention and do not constitute a limitation of the present invention.

[0026] The automatic MTF estimation method based on real-shot images disclosed in the present invention mainly consists of three steps. The image processing technology disclosed in the first two steps is called an automatic target area extraction algorithm, which specifically includes:

[0027] (1) Image edge detection module. In operation 101, 30 pixels around the edges of the actual infrared real-shot image are cut to prevent the target area from crossing the image boundary. In operation 102, the cropped image is subjected to Gaussian low-pass filtering to filter out noise and high-frequency noise. In operation 103, the filtered image is subjected to straight line segment detection using the Line Segment Detector (LSD) algorithm. The effect after processing is as follows: Figure 5 As shown in the figure, since the infrared system has a lot of noise in the pictures taken, the LSD algorithm needs to scale the image to 0.5 times of the original image before detection.

[0028] In operation 104, the line segments detected by the LSD algorithm are restricted in length, angle, and grayscale values ​​on both sides of the line segments. Define the attributes of the axis in the image: let axis=1 for the vertical axis of the image and axis=0 for the horizontal axis. Take the vertical axis of the image (axis=1) as the reference 0 degree, and define the clockwise rotation as 0 to 90 degrees, and define the counterclockwise rotation as 0 to -90 degrees. If the angle of the straight line is in the range of -45 to 45 degrees, the direction of the straight line is defined to be consistent with the vertical direction of the image, and the value of the axis is 1; if the angle of the straight line is in the range of -90 to -45, and the direction of the straight line is consistent with the horizontal direction of the image in the range of 45 to 90 degrees, the value of the axis is 0. The length and angle restrictions are reflected in that the length of the line segment must be greater than the minimum specified length (30 in this example), and the angle of the line segment must be within the specified angle range of -angle to angle (30 in this example). The gray value of the line segment is limited as follows: the line segment with axis = 1 takes a pair of feature points 7 pixels to the left and 7 pixels to the right based on the three trisection points of the line segment. Similarly, the line segment with axis = 0 takes a pair of feature points 7 pixels upward and 7 pixels downward. The gray value conditions are as follows: Figure 4 As shown; when the gray value differences of the image pixels corresponding to the three pairs of feature points are all greater than graf_diff (100 in this example) and meet the length and angle conditions, this step determines that the line segment meets the edge condition and puts it into the edge line segment set.

[0029] (2) Target region search module: In this step, the algorithm first starts a breadth search of the target region, which includes operations 201 and 202, and aims to screen out possible target regions from a large number of regions.

[0030] In operation 201, a line segment in the edge set is obtained, and the detector program is constructed as follows Figure 6The sliding window shown slides along the edge from top to bottom (or from left to right) with a certain step length. The sliding window is rectangular, and the area size (width×height) is determined by the algorithm user, with a minimum of a square area of ​​30×30 pixels. The larger the area, the more pixels it contains, and the better the effect of calculating MTF; but the larger the area, the more noise it contains, and it is difficult for the window to meet the conditions of the target area. The user can choose a suitable window size based on the actual quality of the image. The step size (step) of the window sliding is also an indefinite parameter. The smaller the step size, the less likely it is to miss the ideal target area, but the search volume and program running time will also increase. The step size and area size are preferably multiples of 5. The default step size of the sliding window is 15 pixels. In this embodiment, the area size is 40×40 pixels and the step size is 10 pixels.

[0031] In operation 202, each time the sliding window moves forward a step along a straight line, the target region detector automatically intercepts the window region and divides the region into two parts using the reference edge, and calculates the grayscale value variance for each part. The grayscale value variance of the image is defined as:

[0032]

[0033]

[0034] Among them, z i is the gray value of a pixel, N is the total number of pixels in the area, mean is the average gray value in the area, and stddev is the gray value variance.

[0035] The width of the blade edge of the real image is generally 6 to 8 pixels. The calculation of the gray value variance should avoid adding the blade edge pixels. Therefore, the calculation area is selected as a right-angled trapezoid consisting of three sides of the window edge straight line and one side of the oblique line 3 pixels away from the blade edge, such as Figure 7 As shown. The total grayscale value variance of the window area is the sum of the grayscale value variances of the two right-angle trapezoids. When the total grayscale value variance is less than the standard threshold value stddev, the window area will be selected as a candidate area, and the coordinate information of the window will be stored in the grayscale uniform area set. Similarly, the standard threshold value stddev can also be set by yourself, and the default threshold value is 15. After the sliding window completes the scanning of all elements in the edge set, the program obtains a preliminary candidate target area.

[0036] Next, the algorithm searches for candidate regions and performs secondary refinement on the grayscale uniform region set, including operation 203, operation 204, operation 205, and operation 206, and finally obtains 10 candidate regions.

[0037] In operation 203, a differential operation is performed on each horizontal pixel in the grayscale uniform area to find the centroid of the grayscale value gradient change rate of each row of pixels, obtain a series of grayscale value gradient centroid point sets, and use the least squares method to fit the centroid to obtain the straight line expression of the local edge.

[0038] In operation 204, the total gray value variance of each window is calculated again by analyzing the local edge according to the breadth search step, and the calculation result is called the true gray value variance of the region.

[0039] In operation 205, in order to avoid the appearance of poor edge in the detection results, that is, the situation where outliers appear in the point set of the gray value gradient centroid of the target area, the target area search algorithm adds a restriction condition on the edge expression fitting error, and incorporates this condition into the evaluation system for the comprehensive ranking of candidate areas. For the problem of poor edge areas in the search results, the sorting and optimization method is adopted to solve it. The root mean square error (RMSE) is introduced as a quantitative indicator to measure the quality of the fitting curve. It refers to the observed value The square root of the ratio of the sum of squares of deviations from the true value y to the number of observation points n is expressed mathematically as

[0040]

[0041] In operation 206, let the criterion for ranking candidate regions be STD, then

[0042] STD=stddev·k1+RMSE·k2

[0043] Wherein, stddev is the standard deviation of the true grayscale value of the candidate region, which is calculated in operation 202. k1 and k2 are corresponding weights, and in this embodiment, k1=k2=1.

[0044] Calculate the STD of all candidate regions and arrange them in ascending order according to STD. Take the top ten candidates as the output of the candidate region search process. If there are less than 10 candidate regions, all regions are used as output, and the output regions are arranged in ascending order according to STD.

[0045] The pseudo code of the automatic target region extraction algorithm is shown in Algorithm 1 at the end of the specification.

[0046] After obtaining the candidate area in the image, the knife edge method specified in the ISO12233 standard is used to calculate it and obtain the MTF of the image. Since the automatic target area extraction algorithm only selects the area that meets the calculation characteristics of the knife edge method at the image processing level, and does not verify whether the various physical quantities in the knife edge method calculation process meet the requirements of the method itself, it is also necessary to add a standard ESF (edge ​​spread function) search to the knife edge method to obtain the final target area and the MTF curve of the image.

[0047] (3) Modulation transfer function calculation module. As the last step of the entire algorithm, the modulation transfer function calculation module not only includes the traditional edge method calculation, but also adds a standard ESF search process to obtain a more accurate infrared image MTF. The standard ESF search process includes operations 303 and 304.

[0048] In operation 301, a candidate region is selected, ESF is calculated according to the edge method, and data points affected by interference are filtered out. According to the method specified in ISO12233, the calculation of ESF requires the steps of obtaining the centroid of the gray value change of each row in the region, fitting the edge straight line expression by the least square method, calculating the distance from the pixel to the straight line, projection, 4 times super sampling, etc. Figure 8 shown.

[0049] In operation 302, the data points of the ESF are preprocessed by filtering or the like. In order to avoid the situation where there are too many bins during 4x oversampling and some bins do not contain any pixels, resulting in the ESF value being 0, the algorithm first scans the ESF data from the middle to both sides before taking the derivative of the ESF. If there is a point equal to 0 in the first half, the value of the next ESF is assigned to it; if there is a point equal to 0 in the second half, the value of the previous ESF is assigned to it. In addition, in order to maintain the uniformity of the grayscale values ​​of the pixels on both sides of the blade edge, limiting filtering is performed to the left and right ends at 0.2 times the total length and 0.8 times the total length of the ESF, respectively. If the grayscale value difference between the front and back (or back and front) points is greater than 10, this point is assigned to the average grayscale value of the front (or back) three points.

[0050] In operation 303, a Fermi function curve is fitted to the ESF using the least squares method. The expression of the Fermi function is:

[0051]

[0052] Among them, a i 、b i 、c i , d are parameters to be determined.

[0053] In this embodiment, a first-order Fermi function is used to perform curve fitting of ESF, and the expression of the fitting function is as follows:

[0054]

[0055] Among them, a, b, c, and d are the parameters to be fitted.

[0056] In operation 304, the root mean square errors of the left and right data points of the gray value average area on both sides of the ESF and the fitting curve are calculated respectively. When both errors are less than the threshold ESF_error (default is 5), the area meets the standard ESF condition and the next MTF calculation is allowed. Here, the default width of the edge area is 10 pixels, and the calculation range of the root mean square error on each side is from 20 data points away from the center of the ESF length to the two end points.

[0057] In operation 305, the MTF is calculated for the area that meets the standard ESF condition according to the ISO12233 standard. The final MTF of the infrared system is obtained after the ESF is derived, Fourier transformed, modulo-normalized, and the relationship between ESF and MTF is shown in the following formula:

[0058]

[0059] Where ESF is the edge spread function, LSF is the line spread function, and MTF is the modulation transfer function.

[0060] In operation 306, the MTF data is checked. When there are two or more normalized MTF data points greater than 1, the MTF data is discarded. After the above operation, if there are still one or more candidate areas that meet the requirements, the MTF calculated in the area with the smallest STD (area ranking criterion) is used as the MTF of the infrared system; if all candidate areas do not meet the requirements, the real image has a lot of noise or the infrared system jitters during shooting, and cannot be used as a reference image for evaluating the MTF of the infrared system. It is necessary to replace the photo and perform the above calculation steps again. The pseudo code of the modulation transfer function calculation module is shown in Algorithm 2 at the end of the specification.

[0061] The pictures in this embodiment are actual pictures taken by an airborne infrared system. The final MTF evaluation results are as follows: Fig. 9 , Fig.10 shown.

[0062] The second embodiment in this specification only modifies some parameters in the algorithm, and all other steps are exactly the same as the previous embodiment. After adjusting the parameters, the target area size is 40×40 pixels, the sliding window step is 5 pixels, and the edge detection gray value difference threshold is 50. The image is a city scene image taken by an infrared imaging system, and its MTF evaluation result is as follows Fig.11 , Fig.12 shown.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the algorithm for automatically calculating and estimating MTF may have various modifications and variations. Any modification, replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0064]

[0065]

Claims

1. A method for automatically calculating the modulation transfer function (MTF) of an infrared system based on real-shot images, characterized in that: It includes an automatic target area extraction algorithm and an improved modulation transfer function calculation method, and the steps are: Step (1) an image edge detection module, preprocessing an infrared image, detecting straight line segments in the infrared image, evaluating the length, angle and grayscale values ​​on both sides of the straight line segments, and determining whether the straight line segments meet the edge requirements; a group of segments that meet the edge requirements is an edge set, and a segment in the edge set is an edge segment; Step (2) a target area search module, constructing a sliding window around the edge segment, intercepting an area with a given size and step length, calculating the grayscale value variance and the edge straight line fitting error within the area, and extracting the candidate area based on the grayscale value variance and the edge straight line fitting error within the area; The steps (1) and (2) together constitute the target area automatic extraction algorithm; Step (3) a modulation transfer function calculation module improves the knife edge method steps specified in the international standard ISO12233, and obtains accurate performance parameter MTF of the infrared system from the extracted candidate area; In the step (2), the extraction of candidate regions includes: the gray value variance and the edge straight line fitting error in the region meet the threshold requirements; the regions within the threshold are secondary selected, and the top ten regions with the smallest weighted sum of the gray value variance and the edge straight line fitting error in the region are selected as candidate regions for algorithm detection; the STD of all candidate regions is calculated, and the STD is a criterion for ranking the candidate regions; In the step (3), the specific steps for improving some steps of the knife edge method specified in the international standard ISO12233 are: limiting filtering the edge spread function ESF from the center to both sides; fitting the edge spread function value with a Fermi function, calculating the root mean square error between the fitting function and the stable data on both sides of the edge spread function, and discarding the candidate area with an error greater than a threshold; removing the candidate area with abnormal MTF values; performing derivative, Fourier transform and normalization operations on the ESF of the remaining area to obtain MTF, detecting the MTF data, and discarding the MTF data when there are two or more normalized MTF data points greater than 1; sorting the candidate areas in the step (2) from small to large according to their STD, and taking the MTF calculated for the candidate area with the smallest STD in the candidate area set as the estimated value of the actual MTF of the infrared system.

2. The method for automatically calculating the modulation transfer function (MTF) of an infrared system based on real-shot images according to claim 1, characterized in that: In the step (1), a line segment detector is used to detect the straight line segments in the infrared image. Before detecting the straight line segments in the infrared image, the infrared image is pre-processed by Gaussian filtering to remove noise and reducing the image amplitude by 0.5 times.

3. The method for automatically calculating the modulation transfer function (MTF) of an infrared system based on real-shot images according to claim 1, characterized in that: The step (2) of calculating the gray value variance and edge straight line fitting error in the region includes: 1) Calculate the rate of change of the grayscale value of each row of pixels by using the first-order difference, and obtain the centroid position of the rate of change of the grayscale value of each row of pixels, and fit the centroid of the rate of change of the grayscale value of each row of pixels by using the least squares method to obtain the edge expression of the region; 2) Calculate the grayscale value variance of the pixels on both sides of the edge area respectively. The grayscale value variance in the entire area is equal to the sum of the grayscale value variances of the pixels on both sides; 3) Calculate the root mean square error in the straight line fitting based on the blade edge expression and the actual center of mass coordinates, that is, the blade edge straight line fitting error.

4. The method for automatically calculating the modulation transfer function (MTF) of an infrared system based on real-shot images according to claim 3, characterized in that: The blade edge region refers to a parallelogram region with a width of 6 pixels and a center of the fitted blade edge straight line.