Infrared image scene adaptive correction method and system based on FPGA

Abstract

The present invention provides an FPGA-based method and system for adaptive correction of infrared image scenes. The infrared detector outputs raw image data; the raw image data is subjected to multi-background calibration correction to obtain corrected raw image data; the changing trend of non-uniformity on the infrared detector focal plane caused by changes in specified factors is calculated and reflected, and the true value of the non-uniformity is estimated; the corrected raw image data is further corrected to make the corrected output value approach the true value of the non-uniformity, so that the response characteristics of each detector unit tend to be averaged, the non-uniformity is eliminated, and a secondary correction of the non-uniformity is completed. The present invention solves the problem of the use of the multi-background calibration method. The correction coefficient calculated by the laboratory calibration background is not suitable for the change of non-uniformity caused by changes in temperature, scene environmental radiation, etc., and improves the infrared image non-uniformity correction effect, resulting in better infrared image imaging effect.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared image scene correction, and in particular to an infrared image scene adaptive correction method and system based on FPGA. Background Art

[0002] The technical solution of the prior art one: shutter correction method. Due to the existence of non-uniformity of infrared detectors, shutter correction is usually used to eliminate non-uniformity of infrared images. A shutter piece is installed in front of the detector, and the shutter piece is opened and closed periodically according to the change of focal temperature or timed periodically to collect background. Two-point correction is performed using the newly collected background. The disadvantage of the prior art one is that this shutter correction method requires the installation of a shutter piece at the front end of the detector, which is not conducive to structural installation and design. At the same time, when the shutter correction is performed, that is, the shutter is closed to collect the background image, the image needs to be frozen and cannot be displayed in real time. During the target observation or temperature measurement process, the target is easily lost, and the experience is extremely poor.

[0003] The technical solution of the second existing technology: a multi-background calibration method. In the laboratory's high and low temperature chamber, controlled at different operating temperatures, background acquisition is performed on a uniform scene, and the gain coefficient and offset of each segment are calculated. During use, the gain and offset coefficients of each segment are called according to the different operating temperatures of the detector to perform a two-point calibration. This calibration method can replace shutter calibration. The disadvantage of the second existing technology is that this calibration method only considers the partial non-uniformity changes caused by the drift of the detector's operating temperature. In actual use, there are other factors that cause non-uniformity, such as ambient radiation at different temperatures. When performing multi-background calibration in the laboratory, the detector and the uniform target scene are both inside the high and low temperature chambers, and the background calibration is performed under the conditions of the same temperature and the same ambient radiation. Therefore, in actual use, the detector and the target scene may not be at the same temperature or under the same environmental radiation. For example, in winter, the infrared thermal imaging instrument is exposed to the ambient radiation of room temperature and observes the target outdoors at a low temperature. At this time, the non-uniformity of the imaging is very obvious. The greater the difference between the ambient radiation and temperature during laboratory calibration and the ambient radiation and temperature of the target environment during use, the more significant the non-uniformity. At this time, the background and corresponding gain and offset coefficients calibrated in the laboratory are not suitable for correcting this part of the non-uniformity, resulting in very poor image effect after correction.

[0004] Because the focal plane of an infrared detector is affected by factors such as manufacturing materials and processes, all detection pixels respond differently to the same background or target blackbody radiation in the same environment and temperature, resulting in response inconsistency, a phenomenon known as infrared image non-uniformity. Furthermore, because the ambient temperature and target radiation conditions vary between the calibration process and actual use, the laboratory-calibrated background and corresponding gain and offset coefficients are not suitable for correcting these non-uniformities when observing a target, further contributing to infrared image non-uniformity.

[0005] Chinese invention patent publication number CN114266708A discloses a random-difference-based infrared image scene correction method, which belongs to the field of infrared image processing. The method is capable of simultaneously performing uniform scene recognition calculations and non-uniformity correction processing during infrared image scene correction processing. A uniform scene pixel column set search method is performed based on random differences and with minimization of the resulting image non-uniformity as the objective function, thereby forming an infrared image scene correction method.

[0006] Regarding the aforementioned related technologies, the inventors believe that they cannot be directly used for target observation or temperature measurement during infrared image display. Furthermore, because shutter correction and multi-background calibration methods eliminate the defects and application effects of non-uniformity, it is particularly important to develop a new scene-adaptive correction method that can be applied in engineering applications. Summary of the Invention

[0007] In view of the defects in the prior art, the purpose of the present invention is to provide an FPGA-based infrared image scene adaptive correction method and system.

[0008] According to the present invention, a method for adaptively correcting infrared image scenes based on FPGA is provided, comprising the following steps:

[0009] Raw data output step: the infrared detector outputs raw image data;

[0010] Multi-background calibration correction step: performing multi-background calibration correction on the original image data to obtain corrected original image data;

[0011] The steps for calculating the true value of non-uniformity are as follows: calculating and reflecting the changing trend of the non-uniformity on the focal plane of the infrared detector caused by the change of the specified factors, and estimating the true value of the non-uniformity;

[0012] Secondary non-uniformity correction step: the corrected original image data is further corrected so that the corrected output value approaches the true non-uniformity value, the response characteristics of each detector unit are averaged, the non-uniformity is eliminated, and the secondary non-uniformity correction is completed.

[0013] Preferably, in the non-uniformity true value calculation step, in order to eliminate the influence of non-uniformity on imaging quality, by considering the correlation of the image sequence in the spatial domain and the time series, the weighted aggregation of each grayscale value in the neighborhood in the three-dimensional direction is used as the output of the non-uniformity true value;

[0014] In the image sequence, the space of the corresponding neighborhood is taken on each frame image, and the weighted aggregation of the grayscale of each pixel relative to the central pixel on the current frame is calculated in the neighborhood space. In the weighted aggregation process, if the weights are equal, it is simplified to average weighted aggregation. However, considering the average weighted aggregation, the edge contour details on the image will be blurred. Therefore, the weighted weight of each pixel is calculated based on the similarity between the pixels in the neighborhood. The similarity s j The calculation formula is shown in formula (1), the weighted weight w j The calculation formula is shown in formula (2):

[0015] s j =|I j -I i |, where I j , I i ∈r (1)

[0016]

[0017] Among them, I i is the gray value of the center pixel in the neighborhood, I j is the grayscale value of pixels at other locations in the neighborhood;

[0018] The weight of each pixel relative to the center pixel is calculated within the neighborhood r by using the correlation between image pixels in time series and spatial domain.

[0019] At the same time, in order to highlight the edge contour details on the image, when the current center pixel is at the boundary of the target, the other pixels in the neighborhood are in different areas. The surrounding pixels are compared with the current pixel grayscale value. If the difference is greater than the selected threshold Th, it is determined that the surrounding pixels are in different areas from the current center pixel. Therefore, the multiplicative coefficient of the weight corresponding to the surrounding pixels is set to 0, otherwise, the multiplicative coefficient of the weight corresponding to the surrounding pixels is set to 1;

[0020] The multiplicative coefficient calculation formula of weight is coef j As shown in formula (3):

[0021]

[0022] The final weight calculation formula for each pixel point relative to the center pixel point of the current frame in the neighborhood space is shown in formula (4):

[0023] W j=coef j *w j (4)

[0024] Among them, w j is the weighted weight within the neighborhood, coef j is the multiplicative coefficient corresponding to the weighted weight, W j is the final weighted weight in the neighborhood;

[0025] After the final weight calculation is completed, the weighted aggregation process of the non-uniformity true value estimation is shown in formula (5), formula (6) and formula (7), and the final estimated value of the non-uniformity true value is I′;

[0026] Sum=∑W j *I j (5)

[0027] W=∑W j (6)

[0028]

[0029] Among them, Sum is the product accumulation sum of the corresponding weights of the pixel grayscale at each position in the neighborhood, and W is the final weighted cumulative sum of the pixel grayscale at each position in the neighborhood;

[0030] Non-uniformity true value Y R This is I′ after the estimation in formula (7) is completed.

[0031] Preferably, in the non-uniformity secondary correction step, the response value after the multi-background calibration correction is corrected again to eliminate the response non-uniformity and difference caused by the inconsistency between the calibration and the actual environmental radiation;

[0032] The output value after multi-background calibration is corrected, and the correction model conforms to the linear relationship;

[0033] Let the corrected output value approach the true value Y of the non-uniformity R In the correction process, the corrected response value needs to be approached to the true value of non-uniformity frame by frame, so that the response characteristics of each detector unit tend to be average and the non-uniformity is eliminated;

[0034] The design error function is shown in formula (8):

[0035]

[0036] Among them, n is the number of frames from the first frame to the tth frame, is the true value of the non-uniformity of the pixels at each position on a frame of image, Y i is the pixel output value at the same position after multi-background calibration correction, G iand O i is the linear coefficient in the correction model;

[0037] When the error function value of each pixel at each position in each frame is the smallest, it is the closest time. At this time, G and O are obtained and substituted into the correction model to complete the secondary correction of non-uniformity.

[0038] Among them, G and O represent the linear correction coefficients needed in the secondary correction process.

[0039] Preferably, in the non-uniformity secondary correction step, in order to minimize the error, the least squares estimation method is used to complete the successive approximation calculation.

[0040] Preferably, the correction method further includes a delay calculation step: adapting the alignment relationship of pixels during the calculation process of the FPGA processor to ensure that the pixels correspond to each other in spatial and temporal order.

[0041] Preferably, the correction method further includes an inter-frame management step: managing the reading and writing of image sequence frames and control logic.

[0042] According to the present invention, an FPGA-based infrared image scene adaptive correction system is provided, which includes the following modules:

[0043] Raw data output module: The infrared detector outputs raw image data;

[0044] Multi-background calibration and correction module: performs multi-background calibration and correction on the original image data to obtain the corrected original image data;

[0045] Non-uniformity true value calculation module: calculates and reflects the changing trend of non-uniformity on the focal plane of the infrared detector caused by changes in specified factors, and estimates the non-uniformity true value;

[0046] Non-uniformity secondary correction module: The corrected original image data is further corrected to make the corrected output value close to the true non-uniformity value, so that the response characteristics of each detector unit tend to be average, eliminating non-uniformity and completing the secondary correction of non-uniformity.

[0047] Preferably, in the non-uniformity true value calculation module, in order to eliminate the influence of non-uniformity on imaging quality, by considering the correlation of the image sequence in the spatial domain and the time series, the weighted aggregation of each grayscale value in the neighborhood in the three-dimensional direction is used as the output of the non-uniformity true value;

[0048] In the image sequence, the space of the corresponding neighborhood is taken on each frame image, and the weighted aggregation of the grayscale of each pixel relative to the central pixel on the current frame is calculated in the neighborhood space. In the weighted aggregation process, if the weights are equal, it is simplified to average weighted aggregation. However, considering the average weighted aggregation, the edge contour details on the image will be blurred. Therefore, the weighted weight of each pixel is calculated based on the similarity between the pixels in the neighborhood. The similarity s j The calculation formula is shown in formula (1), the weighted weight w j The calculation formula is shown in formula (2):

[0049] s j =|I j -Ii|, where I j , I i ∈r (1)

[0050]

[0051] Among them, I i is the gray value of the center pixel in the neighborhood, I j is the grayscale value of pixels at other locations in the neighborhood;

[0052] The weight of each pixel relative to the center pixel is calculated within the neighborhood r by using the correlation between image pixels in time series and spatial domain.

[0053] At the same time, in order to highlight the edge contour details on the image, when the current center pixel is at the boundary of the target, the other pixels in the neighborhood are in different areas. The surrounding pixels are compared with the current pixel grayscale value. If the difference is greater than the selected threshold Th, it is determined that the surrounding pixels are in different areas from the current center pixel. Therefore, the multiplicative coefficient of the weight corresponding to the surrounding pixels is set to 0, otherwise, the multiplicative coefficient of the weight corresponding to the surrounding pixels is set to 1;

[0054] The multiplicative coefficient calculation formula of weight is coef j As shown in formula (3):

[0055]

[0056] The final weight calculation formula for each pixel point relative to the center pixel point of the current frame in the neighborhood space is shown in formula (4):

[0057] W j =coef j *w j (4)

[0058] Among them, w j is the weighted weight within the neighborhood, coef j is the multiplicative coefficient corresponding to the weighted weight, Wj is the final weighted weight in the neighborhood;

[0059] After the final weight calculation is completed, the weighted aggregation process of the non-uniformity true value estimation is shown in formula (5), formula (6) and formula (7), and the final estimated value of the non-uniformity true value is I′;

[0060] Sum=∑W j *I j (5)

[0061] W=∑W j (6)

[0062]

[0063] Among them, Sum is the product accumulation sum of the corresponding weights of the pixel grayscale at each position in the neighborhood, and W is the final weighted cumulative sum of the pixel grayscale at each position in the neighborhood;

[0064] Non-uniformity true value Y R This is I′ after the estimation in formula (7) is completed.

[0065] Preferably, in the non-uniformity secondary correction module, the response value after the multi-background calibration correction is corrected again to eliminate the response non-uniformity and difference caused by the inconsistency between the calibration and the actual environmental radiation;

[0066] The output value after multi-background calibration is corrected, and the correction model conforms to the linear relationship;

[0067] Let the corrected output value approach the true value Y of the non-uniformity R In the correction process, the corrected response value needs to be approached to the true value of non-uniformity frame by frame, so that the response characteristics of each detector unit tend to be average and the non-uniformity is eliminated;

[0068] The design error function is shown in formula (8):

[0069]

[0070] Among them, n is the number of frames from the first frame to the tth frame, is the true value of the non-uniformity of the pixels at each position on a frame of image, Y i is the pixel output value at the same position after multi-background calibration correction, G i and O i is the linear coefficient in the correction model;

[0071] When the error function value of each pixel at each position in each frame is the smallest, it is the closest time. At this time, G and O are obtained and substituted into the correction model to complete the secondary correction of non-uniformity.

[0072] Among them, G and O represent the linear correction coefficients needed in the secondary correction process.

[0073] Preferably, in the non-uniformity secondary correction module, in order to minimize the error, the least squares estimation method is used to complete the successive approximation calculation.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] 1. The FPGA-based adaptive infrared image scene correction method of the present invention eliminates a shutter mechanism and corresponding electrical module in the infrared thermal imaging system, solving the user experience problem of freezing the infrared image display due to shutter opening and closing during the infrared thermal imaging display process, resulting in loss of observation targets or temperature measurement targets;

[0076] 2. The present invention also solves the problem of the use of the multi-background calibration method. The correction coefficient calculated by the calibration background in the laboratory is not suitable for the change of non-uniformity caused by changes in temperature, scene environment radiation, etc., thereby improving the infrared image non-uniformity correction effect and making the infrared image imaging effect better;

[0077] 3. Compared with the processing process on traditional PCs (Personal Computers) and other industrial computers, the pipeline (pipeline parallel processing) data processing architecture based on the FPGA chip of this invention reduces power consumption and size, effectively improves processing speed, achieves real-time processing, and reduces processing latency. It can be deployed in various small infrared thermal imaging systems, reducing costs, improving efficiency, and greatly enhancing the user experience.

[0078] 4. Some processing modules of the present invention are designed to be parameterized and modularized, and can be reconfigured and reused without modifying the logic function or code a second time. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0080] Figure 1 is the system block diagram;

[0081] Figure 2 Correcting the distribution map for the detection unit array response;

[0082] Figure 3 Schematic diagram of non-uniform true value estimation;

[0083] Figure 4 The first schematic diagram of the least squares estimation successive approximation;

[0084] Figure 5 The second schematic diagram of the least squares estimation successive approximation;

[0085] Figure 6 Schematic diagram of non-uniform quadratic correction distribution;

[0086] Figure 7 This is a schematic diagram of delay analysis calculation;

[0087] Figure 8 This is a schematic diagram of neighborhood 3 frame management;

[0088] Figure 9 Schematic diagram of N-frame successive approximation iteration. DETAILED DESCRIPTION

[0089] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0090] The embodiment of the present invention discloses an infrared image scene adaptive correction method based on FPGA, such as Figure 1 The figure below shows a block diagram of the system of the present invention. The raw image data output by the infrared sensor first undergoes multi-background calibration to eliminate non-uniformity variations caused by changes in the infrared sensor's operating temperature. It then undergoes a series of processes, including true non-uniformity estimation, a delay module, frame-by-frame iterative approximation, secondary non-uniformity correction calculation, and inter-frame management. Figure 1 In the figure, X represents the raw image data output by the infrared detector; Y represents the corrected raw image data obtained by the multi-background calibration module after processing the raw image data; G / O represents the linear correction coefficient required in the secondary correction process; and Y' represents the output value after the secondary correction.

[0091] The calibration method comprises the following steps:

[0092] Raw data output step: The infrared detector outputs raw image data.

[0093] Multi-background calibration correction step: Perform multi-background calibration correction on the original image data to obtain corrected original image data.

[0094] Specifically, in an infrared focal plane array, although the response function of each detection unit is a nonlinear function, within a relatively small time range, the response curve of the detector unit can be approximated as a straight line. From a time series perspective, the response function of the detection unit can be expressed as a segmented linear response model. However, in the entire detector unit array, the response function of each detection unit is expressed as a response model with different parameters. When observing the uniform radiation of the same target, the output response is inconsistent, indicating the non-uniformity of the detector. After multi-background calibration correction, such as Figure 2 As shown, L1, L2, L3, L4, L5, and L6 represent the column direction, and R1, R2, and R3 represent the row direction. Each small square represents a detection pixel on the detector array. (K1, B1) represents the linear response function with correction parameters K1 and B1, which is Y1 = K1 * X1 + B1. (K2, B2) represents the linear response function with correction parameters K2 and B2, which is Y2 = K2 * X2 + B2. Here, X1 and X2 are the response output values ​​of the infrared detector sensor corresponding to the pixel in each row and column. Theoretically, after multi-background calibration, Y1 and Y2 should be consistent or nearly consistent when observing a uniform target. However, the ambient radiation used in practice differs from that used during laboratory calibration. When observing a uniform background at a certain ambient temperature, the grayscale values ​​Y1, Y2, and Y3 in each row and column direction, output after the aforementioned multi-background calibration, are still inconsistent, indicating response differences and exhibiting very obvious non-uniformity.

[0095] The True Non-Uniformity Calculation Step calculates and reflects the changing trend of non-uniformity on the infrared detector's focal plane due to changes in specified factors, and estimates the true non-uniformity value. This module, known as the True Non-Uniformity Estimation Module, calculates and reflects the changing trend of non-uniformity on the infrared detector's focal plane due to changes in factors such as ambient radiation over time. The estimated true value serves as the basis for subsequent frame-by-frame iterative approximation.

[0096] In order to eliminate the impact of non-uniformity on imaging quality, by considering the correlation of image sequences in the spatial domain and time series, the weighted aggregation of each grayscale value in the neighborhood in the three-dimensional direction is used as the output of the non-uniformity true value. In the image sequence, the space of the corresponding neighborhood is taken on each frame image, and the weighted aggregation of the grayscale of each pixel relative to the central pixel on the current frame is calculated in the neighborhood space. In the weighted aggregation process, if the weights are equal, it is simplified to average weighted aggregation. However, considering the average weighted aggregation, the edge contour details on the image will be blurred. Therefore, the weighted weight of each pixel is calculated based on the similarity between the pixels in the neighborhood. The similarity s j The calculation formula is shown in formula (1), the weighted weight w j The calculation formula is shown in formula (2):

[0097] s j =|I j -I i |, where I j , I i ∈r (1)

[0098]

[0099] Among them, I i is the gray value of the center pixel in the neighborhood, I j is the grayscale value of pixels at other locations in the neighborhood.

[0100] Specifically, in order to better eliminate the impact of this part of non-uniformity on imaging quality, in the present invention, the correlation of the image sequence in the spatial domain and time series is considered, and the weighted aggregation of each gray value in the neighborhood in the 3D direction is used as the output of the non-uniformity true value. Figure 3 As shown, in the image time series, frames 1, 2, and 3 correspond to the current frame, the previous frame, and the previous 2 frames. The corresponding neighborhood 3*3 space is taken on each frame image, and the grayscale of each pixel relative to the center pixel on the current frame is calculated in the domain space of the 3 frames. In the weighted aggregation process, if the weights are equal, it can be simplified to average weighted aggregation. However, considering the average weighted aggregation, the edge contour details on the image will be blurred. Therefore, the weighted weight of each pixel is calculated based on the similarity between the pixels in the neighborhood. Similarity s j The calculation formula is shown in (1), the weighted weight w j The calculation formula is shown in (2).

[0101] In this way, the correlation between image pixels in time series and space domain is used to calculate the weight of each pixel relative to the center pixel in the neighborhood r. The center pixel in the neighborhood will also participate in the calculation, that is,

[0102] s j =0,

[0103] At this time, the weight is the largest, that is,

[0104] w j =1.

[0105] At the same time, in order to highlight the edge contour details on the image, when the current center pixel is located at the boundary of the target, other pixels in the neighborhood are in different areas. The surrounding pixels are compared with the current pixel grayscale value. If the difference is greater than the selected threshold Th (the threshold can be input externally), it is determined that the surrounding pixels are in different areas from the current center pixel. Therefore, the multiplicative coefficient of the weight corresponding to the surrounding pixels is set to 0, otherwise, the multiplicative coefficient of the weight corresponding to the surrounding pixels is set to 1.

[0106] So the multiplicative coefficient calculation formula of weight is coefj As shown in formula (3):

[0107]

[0108] Therefore, the final weighted weight calculation formula for each pixel relative to the center pixel of the current frame in the 3*3*3 neighborhood space composed of the current frame, the previous frame, and the previous two frames is shown in formula (4):

[0109] W j =coef j *w j (4)

[0110] Among them, w j is the weighted weight in the above neighborhood, coef j is the multiplicative coefficient corresponding to the weighted weight, W j is the final weighted weight in the neighborhood.

[0111] After the final weight calculation is completed, the weighted aggregation process of the non-uniformity true value estimation is shown in formula (5), formula (6) and formula (7), and the final estimated value of the non-uniformity true value is I′.

[0112] Sum=∑W j *I j (5)

[0113] W=∑W j (6)

[0114]

[0115] Among them, Sum is the product accumulation sum of the corresponding weights of the pixel grayscale at each position in the neighborhood, and W is the final weighted cumulative sum of the pixel grayscale at each position in the neighborhood. The estimation of the true value of non-uniformity represents the reflection of the response to the radiation of the same uniform scene target under different environmental radiation conditions. At the same time, after the above weighted aggregation, the non-uniformity of the response can be well improved. The detection response output values ​​of all units tend to the average value in a local neighborhood, and the difference between the response values ​​of each detection unit tends to decrease. The true value of non-uniformity Y R This is I′ after the estimation in formula (7) is completed.

[0116] Secondary non-uniformity correction step: the corrected original image data is further corrected so that the corrected output value approaches the true non-uniformity value, the response characteristics of each detector unit are averaged, the non-uniformity is eliminated, and the secondary non-uniformity correction is completed.

[0117] The response value after multi-background calibration is corrected again to eliminate the response non-uniformity and difference caused by the inconsistency between the calibration and the actual environment radiation. That is, the true value of non-uniformity Y R That is I' after the estimation in the above formula (7). Figure 1 The response value after multi-background calibration is corrected again. The purpose of the correction is to eliminate the response non-uniformity and difference caused by the inconsistency between the calibration and the actual environmental radiation.

[0118] The output values ​​after multi-background calibration are corrected, and the correction model conforms to the linear relationship. Specifically, the output values ​​Y1, Y2, Y3...Yn after multi-background calibration are transformed and corrected to Y1', Y2', Y3'...Yn'. The correction model still conforms to the linear relationship, that is,

[0119] Y1'=G1*Y1+O1;

[0120] Y2'=G2*Y2+O2;

[0121] Y3'=G3*Y3+O3.

[0122] It is equivalent to the linear function model Y=K*X+B, G1, G2, and G3 are the gain coefficients in the transformation process, and O1, O2, and O3 are the offset coefficients in the transformation process.

[0123] Let the corrected output values ​​Y1', Y2', and Y3' approach the non-uniform true value Y R Therefore, in the correction process, the corrected response value needs to be approached to the true value of the non-uniformity frame by frame, so that the response characteristics of each detector unit tend to be average and the non-uniformity is eliminated.

[0124] The design error function is shown in formula (8):

[0125]

[0126] Among them, n is the number of frames from the first frame to the tth frame, is the true value of the non-uniformity of the pixels at each position on a frame of image, Y i for Figure 1 The pixel output value of the same position point after multi-background calibration correction, G i and O i is the linear coefficient in the correction model.

[0127] When the error function value of each pixel at each position in each frame is the smallest, it is the closest. At this time, G and O are obtained and substituted into the above correction model to complete the quadratic correction of non-uniformity. Among them, G and O represent the linear correction coefficients needed in the quadratic correction process. In order to minimize the error here, the least squares estimation method is used to complete the calculation of successive approximation. Figure 4 and Figure 5 Shown is a schematic diagram of least squares estimation calculation.

[0128] Figure 4 It means that the center pixel position 0 of the current frame n=1 is the starting point, and the time series frame is traversed forward t frames, that is, the pixel point at position 0 of frame n=2, n=3, 4, 5... the pixel gray value Y of the same position in frame t i and Pairing, find a straight line by successive approximation so that all Y i The sum of the Euclidean distances to this straight line is the minimum, that is, the error function of formula (8) is the minimum. This straight line is the fitting straight line obtained by least squares estimation. The linear coefficients of the straight line equation are G and O. The extreme values ​​can be obtained by expanding formula (8) and taking the derivative with respect to G and O.

[0129] exist Figure 5 In the equation, Yi is the original value of the pixel at each position in a frame of image after multi-background calibration correction. YR is the estimated value of the true value of the non-uniformity of the pixel at the same position in a frame of image.

[0130] Similarly, in the above frame sequence, a set of linear correction coefficients G and O will be obtained at position 1, position 2, position 3, and so on through the entire detector array unit, such as Figure 6 As shown, the above correction model is substituted into the above correction model to complete the secondary correction of non-uniformity.

[0131] Delay calculation step: Adapt the pixel alignment relationship during the FPGA processor calculation process to ensure that the pixels correspond to each other in spatial and temporal order.

[0132] Specifically, the delay calculation module is used to adapt the pixel alignment relationship during the FPGA processor calculation process to ensure that the image pixels are strictly matched in space and time order. FPGA is called Field Programmable Gate Array in English, and its Chinese translation is Field Programmable Gate Array. In terms of time series, because the image sequence input by the sensor takes a certain period of time each time it passes through a processing or calculation module. The analysis and calculation process of delay modules 1 and 2 is as follows: Figure 7As shown in the figure, the first, second, and third frames represent the construction of a 3*3*3 neighborhood space. The true value estimation of the non-uniformity is performed within this neighborhood space. Because the image data sequence is output pixel by pixel in rows and columns in a clock cycle, a delay of 2 rows + 2 clock cycles is required to obtain 3*3*3 = 27 pixels. The weight calculation, frame-by-frame approximation calculation, and calculation of G and O, etc., take a total of 26 clock cycles. Therefore, delay modules 1 and 2 take a total of 2 rows + 28 clock cycles to complete the calculation of the correction model and parameters required for the secondary correction of the first pixel. A pipeline structure is designed inside the FPGA so that multiple pixels participate in the calculation at the same time within these 2 rows + 28 clock cycles. Subsequently, the correction parameters of the second pixel, the third pixel, and so on are calculated in order in the row and column directions. The delay in processing the latter pixel is only one clock cycle longer than that of the previous pixel. This pipeline structure processing method has low delay, high throughput, and fast speed, and is very suitable for parallel computing in the image processing process. Figure 7 In the example, Delay2line+2clk indicates a delay of 2 lines + 2 beats, and Compute Delay 26clk indicates a computation delay of 26 beats.

[0133] Inter-frame management step: manages the reading and writing and control logic of image sequence frames.

[0134] Specifically, the inter-frame management module is used to manage the reading and writing of image sequence frames and the control logic. Figure 8 As shown in the figure, according to the infrared sensor's readout timing logic, the image data video stream is output frame by frame in the row and column directions. During the non-uniformity ground truth estimation, a 3x3 neighborhood space of three frames must be constructed. Therefore, the data of the previous and two previous frames of the current frame must be read from the DDR memory simultaneously. The readout image data stream matches the pixels of the sensor data stream one-to-one to avoid pixel misalignment during neighborhood calculation. Simultaneously, the current frame's image is written to other DDR memory storage spaces. Sequence numbers 0, 1, and 2 in the figure represent pre-allocated storage spaces in the DDR memory. When the current frame's image data stream is written to storage space 2, the images in storage spaces 0 and 1 are simultaneously read. Similarly, as the video sequence progresses, when the current frame is written to storage space 0, the images in storage spaces 1 and 2 are simultaneously read to construct a new three-frame neighborhood. This process continues in this manner, completing neighborhood construction and weight and coefficient calculation for all video frames. DDR represents a memory bank.

[0135] like Figure 9The figure shows a schematic diagram of a K-frame successive approximation iterative calculation. After the true non-uniformity value calculation is completed, G and O are iteratively calculated frame by frame to complete the secondary correction of the infrared image non-uniformity. In the present invention, the number of iterations is set to N = 16 frames. Instead of calculating the correction parameters every 16 frames, the correction parameters are calculated and the secondary non-uniformity correction is performed once for each frame. Initially, G / O is calculated from frames 0 to 15 using least squares estimation. As the video sequence progresses, frames 1 to 16 are used for the second time, and frames 2 to 17 are used for the third time. This iterative calculation is repeated for each frame, ensuring the speed and accuracy of the least squares estimation convergence while also ensuring the continuity of the calculated G and O. This ensures that the corrected infrared image continuously and smoothly responds to changes in the infrared target's radiation without sudden changes.

[0136] The embodiment of the present invention also discloses an infrared image scene adaptive correction system based on FPGA, such as Figure 1 As shown, it includes the following modules:

[0137] Raw data output module: The infrared detector outputs raw image data.

[0138] Multi-background calibration and correction module: performs multi-background calibration and correction on the original image data to obtain the corrected original image data.

[0139] Non-uniformity true value calculation module: calculates and reflects the changing trend of non-uniformity on the focal plane of the infrared detector caused by changes in specified factors, and estimates the non-uniformity true value.

[0140] In order to eliminate the impact of non-uniformity on imaging quality, the correlation of image sequences in spatial and temporal domains is considered, and the weighted aggregation of grayscale values ​​in the neighborhood in the three-dimensional direction is used as the output of the true non-uniformity value.

[0141] In the image sequence, the space of the corresponding neighborhood is taken on each frame image, and the weighted aggregation of the grayscale of each pixel relative to the central pixel on the current frame is calculated in the neighborhood space. In the weighted aggregation process, if the weights are equal, it is simplified to average weighted aggregation. However, considering the average weighted aggregation, the edge contour details on the image will be blurred. Therefore, the weighted weight of each pixel is calculated based on the similarity between the pixels in the neighborhood. The similarity s j The calculation formula is shown in formula (1), the weighted weight w j The calculation formula is shown in formula (2):

[0142] s j =|I j -I i |, where I j , I i ∈r (1)

[0143]

[0144] Among them, I i is the gray value of the center pixel in the neighborhood, I j is the grayscale value of pixels at other locations in the neighborhood.

[0145] The correlation between image pixels in time series and spatial domain is used to calculate the weight of each pixel relative to the central pixel in the neighborhood r.

[0146] At the same time, in order to highlight the edge contour details on the image, when the current center pixel is located at the boundary of the target, other pixels in the neighborhood are in different areas. The surrounding pixels are compared with the current pixel grayscale value. If the difference is greater than the selected threshold Th, it is determined that it is in a different area from the current center pixel. Therefore, the multiplicative coefficient of the weight is set to 0, otherwise, its weight multiplicative coefficient is set to 1.

[0147] The multiplicative coefficient calculation formula of weight is coef j As shown in formula (3):

[0148]

[0149] The final weight calculation formula for each pixel point relative to the center pixel point of the current frame in the neighborhood space is shown in formula (4):

[0150] W j =coef j *w j (4)

[0151] Among them, w j is the weighted weight within the neighborhood, coef j is the multiplicative coefficient corresponding to the weighted weight, W j is the final weighted weight in the neighborhood.

[0152] After the final weight calculation is completed, the weighted aggregation process of the non-uniformity true value estimation is shown in formula (5), formula (6) and formula (7), and the final estimated value of the non-uniformity true value is I′.

[0153] Sum=∑W j *I j (5)

[0154] W=∑W j (6)

[0155]

[0156] Among them, Sum is the product accumulation sum of the weights corresponding to the pixel grayscale of each position in the neighborhood, and W is the final weighted cumulative sum of the pixel grayscale of each position in the neighborhood.

[0157] Non-uniformity true value Y R This is I′ after the estimation in formula (7) is completed.

[0158] Non-uniformity secondary correction module: The corrected original image data is further corrected to make the corrected output value close to the true non-uniformity value, so that the response characteristics of each detector unit tend to be average, eliminating non-uniformity and completing the secondary correction of non-uniformity.

[0159] The response value after multi-background calibration correction is corrected again to eliminate the response non-uniformity and difference caused by the inconsistency between the calibration and the actual environmental radiation.

[0160] The output values ​​after multi-background calibration correction are corrected, and the correction model conforms to the linear relationship.

[0161] Let the corrected output value approach the true value Y of the non-uniformity R In the correction process, the corrected response value needs to be approached to the true value of the non-uniformity frame by frame, so that the response characteristics of each detector unit tend to be average and the non-uniformity is eliminated.

[0162] The design error function is shown in formula (8):

[0163]

[0164] Among them, n is the number of frames from the first frame to the tth frame, is the true value of the non-uniformity of the pixels at each position on a frame of image, Y i is the pixel output value at the same position after multi-background calibration correction, G i and O i is the linear coefficient in the correction model.

[0165] When the error function value for each pixel at each position in each frame is minimized, the approximation is achieved. At this point, G and O are calculated and substituted into the correction model to complete the quadratic correction of the non-uniformity. G and O represent the linear correction coefficients used in the quadratic correction process. To minimize the error, the least squares estimation method is used to perform the successive approximation calculations.

[0166] The present invention provides an FPGA-based infrared image scene adaptive correction method, a neighborhood weight calculation method, a non-uniformity true value estimation method, and a successive approximation iterative calculation method and process for correction coefficients.

[0167] In engineering applications, infrared thermal imaging cores must meet the requirements of low power consumption, low cost, excellent imaging quality, and clear image contrast. FPGA processors are typically used to complete the acquisition, correction, and image processing of infrared thermal images. This scene-based adaptive correction effectively eliminates image degradation caused by component non-uniformity while eliminating a shutter mechanism and electrical module, significantly improving the user experience.

[0168] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0169] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for adaptive correction of infrared image scenes based on FPGA, characterized in that: The steps include: Raw data output step: the infrared detector outputs raw image data; Multi-background calibration correction step: performing multi-background calibration correction on the original image data to obtain corrected original image data; The steps for calculating the true value of non-uniformity are as follows: calculating and reflecting the changing trend of the non-uniformity on the focal plane of the infrared detector caused by the change of the specified factors, and estimating the true value of the non-uniformity; Secondary non-uniformity correction step: the corrected original image data is further corrected to make the corrected output value close to the true non-uniformity value, so that the response characteristics of each detector unit tend to be averaged, eliminating the non-uniformity and completing the secondary non-uniformity correction; In the step of calculating the true non-uniformity value, in order to eliminate the influence of non-uniformity on imaging quality, the correlation of the image sequence in the spatial domain and the time series is considered, and the weighted aggregation of the grayscale values ​​in the neighborhood in the three-dimensional direction is used as the output of the true non-uniformity value; In the image sequence, the space of the corresponding neighborhood is taken on each frame image, and the grayscale of each pixel relative to the center pixel on the current frame is calculated in the neighborhood space. In the weighted aggregation process, if the weights are equal, it is simplified to average weighted aggregation. However, considering the average weighted aggregation, the edge contour details on the image will be blurred. Therefore, the weighted weight of each pixel is calculated based on the similarity between the pixels in the neighborhood. Similarity The calculation formula is shown in formula (1), weighted weight The calculation formula is shown in formula (2): ,in , ∈ r (1) （2） in, is the gray value of the center pixel in the neighborhood, is the grayscale value of pixels at other locations in the neighborhood; Using the correlation of image pixels in time series and spatial domain, r Calculate the weight of each pixel relative to the center pixel; At the same time, in order to highlight the edge contour details on the image, when the current center pixel is at the boundary of the target, the other pixels in the neighborhood are in different areas, and the surrounding pixels are compared with the current pixel grayscale value. If the difference is greater than the selected threshold T h , it is determined that the surrounding pixels are in different areas from the current center pixel, so the multiplicative coefficients of the weights corresponding to the surrounding pixels are set to 0, otherwise, the multiplicative coefficients of the weights corresponding to the surrounding pixels are set to 1; The formula for calculating the multiplicative coefficient of weight As shown in formula (3): = （3） The final weighted calculation formula for each pixel point relative to the center pixel point of the current frame in the neighborhood space is shown in formula (4): * （4） in, is the weighted weight within the neighborhood, is the multiplicative coefficient corresponding to the weighted weight, is the final weighted weight in the neighborhood; After the final weight calculation is completed, the weighted aggregation process of the non-uniformity true value estimation is shown in formula (5), formula (6) and formula (7). The final estimate of the non-uniformity true value is ; Sum= (5) W= （6） = （7） Among them, Sum is the product accumulation sum of the corresponding weights of the pixel grayscale at each position in the neighborhood, and W is the final weighted cumulative sum of the pixel grayscale at each position in the neighborhood; True value of non-uniformity That is, after the estimation in formula (7) is completed .

2. The FPGA-based infrared image scene adaptive correction method according to claim 1, characterized in that: In the non-uniformity secondary correction step, the response value after the multi-background calibration correction is corrected again to eliminate the response non-uniformity and difference caused by the inconsistency between the calibration and the actual environmental radiation; The output value after multi-background calibration is corrected, and the correction model conforms to the linear relationship; Let the corrected output value approach the true value of non-uniformity In the correction process, the corrected response value needs to be approached to the true value of non-uniformity frame by frame, so that the response characteristics of each detector unit tend to be average and the non-uniformity is eliminated; The design error function is shown in formula (8): （8） in, n From the first frame to the t The frames gradually approach, is the true value of the non-uniformity of the pixels at each position on a frame of image, is the pixel output value at the same position after multi-background calibration correction, is the linear coefficient in the correction model; When the error function value of each pixel at each position in each frame is the smallest, it is the closest time. At this time, G and O are obtained and substituted into the correction model to complete the secondary correction of non-uniformity. Among them, G and O represent the linear correction coefficients needed in the secondary correction process.

3. The FPGA-based infrared image scene adaptive correction method according to claim 2, characterized in that: In the step of secondary correction of non-uniformity, in order to minimize the error, the least squares estimation method is used to complete the calculation of successive approximation.

4. The FPGA-based infrared image scene adaptive correction method according to claim 1, characterized in that: The correction method also includes a delay calculation step: adapting the alignment relationship of pixels during the calculation process of the FPGA processor to ensure that the pixels correspond to each other in spatial and temporal order.

5. The FPGA-based infrared image scene adaptive correction method according to claim 1, characterized in that: The correction method also includes an inter-frame management step: managing the reading and writing of image sequence frames and controlling logic.

6. An FPGA-based infrared image scene adaptive correction system, characterized in that: Includes the following modules: Raw data output module: The infrared detector outputs raw image data; Multi-background calibration and correction module: performs multi-background calibration and correction on the original image data to obtain the corrected original image data; Non-uniformity true value calculation module: calculates and reflects the changing trend of non-uniformity on the focal plane of the infrared detector caused by changes in specified factors, and estimates the non-uniformity true value; Non-uniformity secondary correction module: This module further corrects the corrected original image data to make the corrected output value approach the true non-uniformity value, average the response characteristics of each detector unit, eliminate non-uniformity, and complete the secondary correction of non-uniformity. In the non-uniformity true value calculation module, in order to eliminate the impact of non-uniformity on imaging quality, by considering the correlation of the image sequence in the spatial domain and time series, the weighted aggregation of each grayscale value in the neighborhood in the three-dimensional direction is used as the output of the non-uniformity true value; In the image sequence, the space of the corresponding neighborhood is taken on each frame image, and the grayscale of each pixel relative to the center pixel on the current frame is calculated in the neighborhood space. In the weighted aggregation process, if the weights are equal, it is simplified to average weighted aggregation. However, considering the average weighted aggregation, the edge contour details on the image will be blurred. Therefore, the weighted weight of each pixel is calculated based on the similarity between the pixels in the neighborhood. Similarity The calculation formula is shown in formula (1), weighted weight The calculation formula is shown in formula (2): ,in , ∈ r (1) （2） in, is the gray value of the center pixel in the neighborhood, is the grayscale value of pixels at other locations in the neighborhood; Using the correlation of image pixels in time series and spatial domain, r Calculate the weight of each pixel relative to the center pixel; At the same time, in order to highlight the edge contour details on the image, when the current center pixel is at the boundary of the target, the other pixels in the neighborhood are in different areas, and the surrounding pixels are compared with the current pixel grayscale value. If the difference is greater than the selected threshold T h , it is determined that the surrounding pixels are in different areas from the current center pixel, so the multiplicative coefficients of the weights corresponding to the surrounding pixels are set to 0, otherwise, the multiplicative coefficients of the weights corresponding to the surrounding pixels are set to 1; The formula for calculating the multiplicative coefficient of weight As shown in formula (3): = （3） The final weighted calculation formula for each pixel point relative to the center pixel point of the current frame in the neighborhood space is shown in formula (4): * （4） in, is the weighted weight within the neighborhood, is the multiplicative coefficient corresponding to the weighted weight, is the final weighted weight in the neighborhood; After the final weight calculation is completed, the weighted aggregation process of the non-uniformity true value estimation is shown in formula (5), formula (6) and formula (7). The final estimate of the non-uniformity true value is ; Sum= (5) W= （6） = （7） Among them, Sum is the product accumulation sum of the corresponding weights of the pixel grayscale at each position in the neighborhood, and W is the final weighted cumulative sum of the pixel grayscale at each position in the neighborhood; True value of non-uniformity That is, after the estimation in formula (7) is completed .

7. The FPGA-based infrared image scene adaptive correction system according to claim 6, characterized in that: In the non-uniformity secondary correction module, the response value after multi-background calibration is corrected again to eliminate the response non-uniformity and difference caused by the inconsistency between the calibration and the actual environmental radiation; The output value after multi-background calibration is corrected, and the correction model conforms to the linear relationship; Let the corrected output value approach the true value of non-uniformity In the correction process, the corrected response value needs to be approached to the true value of non-uniformity frame by frame, so that the response characteristics of each detector unit tend to be average and the non-uniformity is eliminated; The design error function is shown in formula (8): （8） in, n From the first frame to the t The frames gradually approach, is the true value of the non-uniformity of the pixels at each position on a frame of image, is the pixel output value at the same position after multi-background calibration correction, is the linear coefficient in the correction model; When the error function value of each pixel at each position in each frame is the smallest, it is the closest time. At this time, G and O are obtained and substituted into the correction model to complete the secondary correction of non-uniformity. Among them, G and O represent the linear correction coefficients needed in the secondary correction process.

8. The FPGA-based infrared image scene adaptive correction system according to claim 7, characterized in that: In the non-uniformity secondary correction module, in order to minimize the error, the least squares estimation method is used to complete the successive approximation calculation.

Images