Method and device for enhancing details of infrared imaging system by combining FPGA (Field Programmable Gate Array)

Abstract

The invention discloses a method and a device for enhancing details of an infrared imaging system in combination with an FPGA (Field Programmable Gate Array). The method comprises the following steps of: 1, performing stripe non-uniformity correction on an infrared image on an FPGA level; step 2, when a hardware analog signal is valued, denoising is carried out; 3, performing short exposure to obtain a first infrared image; 4, Fourier transform is carried out on the first infrared image obtained in the step 3, and high frequency is reserved; 5, resetting the exposure value according to the judgment index to obtain a second infrared image; step 6, Fourier transform is carried out on the second infrared image obtained in the step 5, and low frequency is reserved; step 7, fusing the high-frequency part of the first infrared image obtained in the step 4 and the low-frequency part of the second infrared image obtained in the step 6; and step 8, performing detail enhancement on the digital image fused in the step 7.

Description

Technical Field

[0001] The invention belongs to the technical field of infrared image correction, and in particular relates to a method and a device for enhancing details of an infrared imaging system in combination with FPGA. Background Art

[0002] Infrared imaging systems are used in many fields, such as military, aerospace, and medical fields. However, due to defects in device manufacturing materials and manufacturing processes, the original infrared images collected by infrared detectors have problems such as poor detail resolution, low contrast, and high noise, and cannot be directly used in engineering practice.

[0003] In infrared imaging systems, image clarity and detail enhancement are key factors affecting system performance. Traditional infrared imaging systems are often affected by noise, non-uniformity, and lighting conditions, resulting in reduced image quality. With the development of FPGA technology, applying it to infrared imaging systems can achieve more efficient image processing capabilities.

[0004] For example, an invention patent with a publication number of CN103530876A discloses an infrared image analysis method based on Fourier transform, which is mainly used for infrared radiation intensity image and polarization image analysis. The analysis method includes: first, Fourier transform the infrared radiation intensity image and infrared polarization image to be analyzed to obtain their respective spectra. Then, the obtained spectrum is centered to obtain a centralized spectrum. Then, according to the created partition model and different n values, the spectral characteristics of the radiation intensity image and the polarization image are analyzed. The experimental results show that this method can better explain the spectral distribution characteristics of infrared images and can provide a theoretical basis for further processing of infrared images.

[0005] Another example is an invention patent with a publication number of CN114066759A, which discloses a real-time distortion correction method and system for infrared images based on FPGA. The correction method includes collecting infrared imaging data of the target to obtain original infrared distortion image data; according to the selected infrared imaging lens, obtaining the distortion correction parameters corresponding to the infrared imaging lens from the dedistortion reverse mapping address data corresponding to the distortion correction parameters of multiple groups of infrared imaging lenses stored in advance; filtering and inter-frame noise reduction processing are performed on the original infrared distortion image data; according to the obtained distortion correction parameters, image correction is performed on the infrared image data after filtering and inter-frame noise reduction processing. This invention not only improves the reusability of the same infrared image distortion correction system, but also ensures the distortion correction effect of the infrared image.

[0006] Another example is an invention patent with a publication number of CN113888442A, which discloses an infrared image non-uniformity correction method and system based on FPGA, including the following steps: Step S1: Obtain the cumulative histogram value of the original image; Step S2: Calculate the new value based on the cumulative histogram value; Step S3: Map the new value in combination with the original image to obtain the final image. This invention patent is based on the special structure of FPGA and the traditional median histogram algorithm, and proposes an infrared image non-uniformity correction method based on FPGA, which can effectively remove the stripe noise on the original infrared image and is easy to implement on FPGA.

[0007] The above-mentioned prior art has the following disadvantages: insufficient anti-noise capability and limitations of algorithm implementation. Summary of the invention

[0008] A brief summary of embodiments of the present invention is provided below to provide a basic understanding of certain aspects of the present invention. It should be understood that the following summary is not an exhaustive summary of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to provide certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0009] In the prior art, there is no solution for processing infrared imaging at the FPGA level, which has technical problems such as slow speed and insufficient details of a single image. The solution for infrared image correction based on FPGA has insufficient noise resistance and limitations in algorithm implementation. Therefore, the present application provides a method for enhancing the details of an infrared imaging system by combining FPGA, which enhances the details of infrared images through stripe non-uniformity correction, denoising, frequency domain processing and fusion of multiple exposure images. Among them, the present application adopts the stochastic gradient descent method to adaptively find the optimal solution, improves the adaptability to complex noise scenes, and enhances the adaptability of stripe removal.

[0010] According to one aspect of the present application, a method for enhancing details of an infrared imaging system by combining an FPGA is provided, comprising: Step 1: Perform stripe non-uniformity correction on the infrared image at the FPGA level; Step 2: De-noising when taking the hardware analog signal value; Step 3: short exposure to obtain a first infrared image; Step 4, performing Fourier transform on the first infrared image to retain high frequencies; Step 5: reset the exposure value according to the judgment index to obtain a second infrared image; Step 6, performing Fourier transform on the second infrared image, retaining the low frequency; Step 7, fusing the high-frequency portion of the first infrared image obtained in step 4 and the low-frequency portion of the second infrared image obtained in step 6; Step 8: Enhance the details of the digital image fused in step 7.

[0011] In step 1, stripe non-uniformity correction is performed at the FPGA level, specifically including: The infrared image with stripe non-uniform noise is expressed as: ; In the above formula, represents pixel coordinates, and represent the image with vertical stripes and the ideal image respectively, Represents vertical stripe noise; the n in An is used to distinguish it from A. An is a whole and n has no physical meaning.

[0012] The above infrared image is subjected to fringe non-uniformity correction. The correction algorithm steps are as follows: Step 1: Rearrange the pixel values ​​of each column in the infrared image so that the pixel values ​​of each column are arranged in the smallest, largest, second smallest, and second largest order. The image obtained by this arrangement is recorded as C. Step 2: Optimize parameters by minimizing: ; Where i represents the row coordinate of the pixel coordinate, D and E represent the correction parameter and constant respectively; Represents the pixel in the i-th row of image C; is the objective function, which represents the error of the correction parameter D to the vertical stripes; Step 3: Use the gradient descent method to iterate each pixel, starting from the first row to the last row, as shown below: ; ; Represents the learning rate. Adjusting its value can be used to control the convergence speed. Represents the gradient direction of the objective function with respect to D; Indicates the error of the correction parameter D to the vertical stripes in the i-th row of pixels in image C; Indicates +1 iteration of calibration parameters, No. The calibration parameters of the iterations are: An represents an image containing vertical stripes; the number of rows of the matrix is ​​the number of rows of the image An. Since the vertical stripe noise is the same in each column, the same calibration parameters can be added to each row.

[0013] Step 2: De-noise the hardware analog signal by median filtering: Given a signal sequence, use a window, sort the values ​​in the window by size, select the middle value (i.e. the median after sorting) as the filter output, and slide the window until the entire signal sequence is traversed.

[0014] Step 3: Collect the first infrared image by setting a shorter exposure time; short exposure helps to avoid overexposure of high brightness areas while capturing details in the image.

[0015] Step 4: Perform Fourier transform on the first infrared image obtained in step 3 to retain high frequencies.

[0016] In this step, the first infrared image obtained from step 3 is subjected to Fourier transform to convert the image from the spatial domain to the frequency domain. In the frequency domain, the high-frequency components of the image mainly contain the edge and detail information of the image. By extracting and retaining these high-frequency components, the detail expression of the image can be enhanced, providing a basis for subsequent image processing steps.

[0017] Fourier transform is a mathematical transformation that converts a signal from the time or space domain to the frequency domain. For a two-dimensional image f(x,y), its Discrete Fourier Transform (DFT) can be expressed as: ; Where: f(x,y) is the original image in the spatial domain, F(u,v) is the Fourier coefficient in the frequency domain; M and N are the width and height of the image respectively; u and v are the coordinates in the frequency domain; The frequency index u is normalized to be in the range of [0,1]. can be considered as a phase change corresponding to a periodic signal; The frequency index v is normalized to be in the range [0,1]. can be considered as a phase change corresponding to a periodic signal; It describes the phase rotation of the signal at each point (x, y) in the spatial domain corresponding to a specific frequency (u, v) in the frequency domain.

[0018] The Inverse Discrete Fourier Transform (IDFT) is used to convert frequency domain data back to the spatial domain: ; In the frequency domain image F(u,v), the part close to the center of the frequency domain is the low-frequency component, and the part closer to the edge is the high-frequency component. In order to enhance the details of the image, it is necessary to retain the high-frequency component. The specific operation is: Set a frequency threshold Threshold, set the frequency components below the threshold to zero, and only retain the frequency components above the frequency threshold, forming a new frequency domain image that retains the high-frequency components ,in: ; For the frequency domain image that retains the high frequency components Perform an inverse Fourier transform (IDFT) to convert it back to the spatial domain to obtain a high-frequency detail image This image mainly shows edges and details, with less overall brightness information. The high-frequency components of the first infrared image are extracted and enhanced, providing the necessary basis for subsequent image fusion and overall detail enhancement.

[0019] Step 5: reset the exposure value according to the judgment index to obtain a second infrared image; Compared with step 3, this time the exposure time is different, which is used to capture details of different brightness areas. According to the preset image quality indicator "global variance of the image", the details and noise level of the current image (first infrared image) are judged. The global variance of the image is calculated, and the exposure parameters are adjusted according to the size of the global variance. The second infrared image is obtained again. The specific formula for calculating the global variance s of the image is as follows: ; Where s is the global variance of the first infrared image; f(x+1,y) is the value of the (x+1,y) pixel of the first infrared image, where f(x,y) is the value of the (x,y) pixel of the first infrared image, where f(x,y+1) is the value of the (x,y+1) pixel of the first infrared image, N x is the number of pixels of the image on the x-axis, N y is the number of pixels of the image on the y-axis.

[0020] The new exposure parameter EX (in milliseconds) is calculated as follows: ; ; Where ra is the new exposure parameter calculated from the global variance of the over-exposure image and the global variance of the short-exposure image, and s tis the average value of the global variance of multiple over-exposure pictures collected by the instrument. For example, the average value of the global variance of multiple over-exposure pictures collected by the instrument used in this application is 1108869, then s t The value is 1108869, which is a specific value for this instrument after experiments. t is the image quality of the overexposed image. We can collect multiple overexposed images and calculate the average value of the global variance to get s t .

[0021] Step 6, performing Fourier transform on the second infrared image, retaining the low frequency; Perform Fourier transform on the second infrared image to extract the low-frequency components of the image. The low-frequency components mainly reflect the overall brightness and structural information of the image. This step is similar to Step 4. Set a frequency threshold Threshold, set the frequency components above the threshold to zero, and only keep the frequency components below the threshold. Form a new frequency domain image .

[0022] Step 7, fusing the high-frequency portion of the first infrared image obtained in step 4 and the low-frequency portion of the second infrared image obtained in step 6; This step fuses the high-frequency components of the first infrared image extracted in step 3 with the low-frequency components of the second infrared image extracted in step 6 to generate an infrared image with rich details and uniform brightness. The fused image retains the details of the highlight area and fully displays the structure and brightness information of the overall image.

[0023] Use linear weighted fusion. is the high frequency component of the first infrared image, is the low-frequency component of the second infrared image, and the fused image It is expressed as: ; Among them, α and β are fusion weight coefficients, satisfying α+β=1. In order to pay more attention to the details of the image, a larger α value is selected.

[0024] Step 8: Digital image detail enhancement: After the image fusion is completed, in order to further improve the clarity and detail expression of the image, nonlinear detail enhancement techniques such as adaptive histogram equalization (CLAHE) and logarithmic transformation are used to process the fused image. Adaptive histogram equalization helps to enhance the local contrast of the image, while logarithmic transformation can effectively enhance the details of the low-brightness area in the image. The combination of these two methods can significantly improve the visual quality of the final image.

[0025] Color adaptive histogram equalization (CLAHE) is a technique for histogram equalization based on local areas (tiles). It can avoid the noise amplification and over-enhancement problems caused by traditional global histogram equalization and is particularly suitable for detail enhancement.

[0026] The specific steps are as follows: 1. The fused image It is divided into several small sub-areas, called tiles. Each tile is usually 16×16 pixels in size.

[0027] 2. Calculate the histogram for each tile separately and perform histogram equalization on it to make the grayscale distribution in the area more uniform. The equalized image will have better local contrast and highlight the details.

[0028] 3. During the histogram equalization process of each tile, contrast limiting is applied to avoid noise amplification due to over-equalization. Contrast limiting is usually implemented by setting a threshold, and the histogram cumulative values ​​exceeding this threshold will be clipped and evenly distributed to the rest of the histogram.

[0029] 4. After processing each tile, use bilinear interpolation to smooth the boundaries between adjacent tiles to avoid block effects, thereby ensuring the smoothness and consistency of the overall image. The final image is .

[0030] Logarithmic transformation is a commonly used nonlinear enhancement technology, which is mainly used to enhance the details of low-brightness areas in images and increase the contrast of these areas. Since logarithmic transformation can compress the brightness range, it has a significant effect on enhancing the dark details in images.

[0031] Adaptive histogram equalization of the image After logarithmic transformation, the transformed image g(x,y) can be expressed as: ; Among them, c is a constant used to adjust the amplitude of the logarithmic transformation, and its value is usually: ; Where L is the grayscale level of the image (for a 12-bit image, L=4096), and M is the maximum grayscale value of the original image. The grayscale value range of the image after logarithmic transformation is usually small. In order to make the image display better, the grayscale of the transformed image can be stretched to map its grayscale value to the entire display range. .

[0032] In the process of detail enhancement, noise may be introduced or amplified. Therefore, in the last step, a bilateral filter or other smoothing technique is used to control the noise of the image, retaining the details while suppressing unnecessary noise.

[0033] According to another aspect of the present application, a device for enhancing the details of an infrared imaging system by combining with FPGA is provided, which is used to implement the steps of the method for enhancing the details of an infrared imaging system by combining with FPGA. Specifically, it includes: The non-uniformity correction module is used to perform stripe non-uniformity correction on infrared images at the FPGA level; De-noising module, used to remove noise when hardware analog signal is taken; A first infrared image acquisition module, used to obtain a first infrared image through short exposure; A high-frequency acquisition module, used for performing Fourier transform on the obtained first infrared image to retain the high-frequency part; A second infrared image acquisition module, used to reset the exposure value according to the judgment index to obtain a second infrared image; A low-frequency acquisition module, used for performing Fourier transform on the second infrared image and retaining the low-frequency part; A fusion module, used for fusing the high frequency part of the first infrared image with the low frequency part of the second infrared image; A detail enhancement module is used to enhance the details of the fused digital image; Among them, in the non-uniformity correction module, stripe non-uniformity correction is performed at the FPGA level, specifically including: The infrared image with stripe non-uniform noise is expressed as: ; In the above formula, represents pixel coordinates, and represent the image with vertical stripes and the ideal image respectively, represents vertical stripe noise; The above infrared image is subjected to fringe non-uniformity correction. The correction algorithm steps are as follows: Step 1: Rearrange the pixel values ​​of each column in the infrared image so that the pixel values ​​of each column are arranged in the smallest, largest, second smallest, and second largest order. The image obtained by this arrangement is recorded as C. Step 2: Optimize parameters by minimizing: ; Where i represents the row coordinate of the pixel coordinate, D and E represent the correction parameter and constant respectively; Represents the pixel in the i-th row of image C; is the objective function, which represents the error of the correction parameter D to the vertical stripes; Step 3: Use the gradient descent method to iterate each pixel, starting from the first row to the last row, as shown below: ; ; Represents the learning rate. Adjusting its value can be used to control the convergence speed. Represents the gradient direction of the objective function with respect to D; Indicates the error of the correction parameter D to the vertical stripes in the i-th row of pixels in image C; Indicates +1 iteration of calibration parameters, No. The calibration parameters of the iterations are: ; An represents an image containing vertical stripes; the number of rows of the matrix is ​​the number of rows of the image An.

[0034] The present application realizes a method and device for enhancing the details of an infrared imaging system in combination with FPGA through the above-mentioned scheme, which has good noise resistance. The infrared image correction scheme based on FPGA in the prior art has insufficient noise resistance and limitations in algorithm implementation. The present application realizes detail enhancement of infrared images through stripe non-uniformity correction, denoising, frequency domain processing and fusion of multiple exposure images. At the same time, the present application adopts the stochastic gradient descent method to adaptively find the optimal solution, improves the adaptability to complex noise scenes, enhances the adaptability of stripe removal, and realizes a more efficient infrared image processing capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention may be better understood by referring to the following description given in conjunction with the accompanying drawings, wherein the same or similar reference numerals are used throughout the drawings to represent the same or similar components. The accompanying drawings, together with the following detailed description, are included in and form a part of this specification and are used to further illustrate the preferred embodiments of the present invention and to explain the principles and advantages of the present invention. In the drawings: Figure 1 This is a flow chart of the method for enhancing the details of an infrared imaging system in combination with FPGA in this application. DETAILED DESCRIPTION

[0036] Embodiments of the present invention will be described below with reference to the accompanying drawings. Elements and features described in one drawing or one embodiment of the present invention may be combined with elements and features shown in one or more other drawings or embodiments. It should be noted that for the purpose of clarity, the representation and description of components and processes that are not related to the present invention and are known to those of ordinary skill in the art are omitted in the drawings and descriptions.

[0037] In infrared imaging systems, image clarity and detail enhancement are key factors affecting system performance. Traditional infrared imaging systems are usually affected by noise, non-uniformity, and lighting conditions, resulting in reduced image quality. With the development of FPGA technology, its application in infrared imaging systems can achieve more efficient image processing capabilities. This paper proposes an infrared imaging detail enhancement method based on FPGA, which achieves detail enhancement of infrared images through stripe non-uniformity correction, denoising, frequency domain processing, and fusion of multiple exposure images. Figure 1 , the steps are as follows: Step 1: Perform stripe non-uniformity correction at the FPGA level; Step 2: De-noising when taking the hardware analog signal value; Step 3: short exposure to obtain a first infrared image; Step 4, performing Fourier transform on the first infrared image to retain its high frequency part; Step 5: reset the exposure value according to the judgment index to obtain a second infrared image; Step 6: Perform Fourier transform on the second infrared image to retain its low-frequency part; Step 7, fusing the high-frequency portion of the first infrared image obtained in step 4 and the low-frequency portion of the second infrared image obtained in step 6; Step 8: Enhance digital image details.

[0038] Among them, step 1, perform stripe non-uniformity correction at the FPGA level: Due to the influence of manufacturing process and use environment, the non-uniformity problem of infrared imaging system still exists. The most obvious manifestation in the image is fixed pattern noise, which greatly reduces the image quality and even makes the infrared detector lose its detection ability. In engineering applications, the corresponding non-uniformity correction technology is the key image preprocessing technology.

[0039] The infrared image with stripe non-uniform noise is expressed as: ; In the above formula, represents pixel coordinates, and represent the image with vertical stripes and the ideal image respectively, represents vertical stripe noise; The above infrared image is subjected to fringe non-uniformity correction. The correction algorithm steps are as follows: Step 1: Rearrange the pixel values ​​of each column in the infrared image so that the pixel values ​​of each column are arranged in the smallest, largest, second smallest, and second largest order. The image obtained by this arrangement is recorded as C. Step 2: Optimize parameters by minimizing: ; Where i represents the row coordinate of the pixel coordinate, D and E represent the correction parameter and constant respectively; Represents the pixel in the i-th row of image C; is the objective function, which represents the error of the correction parameter D to the vertical stripes; Step 3: Use the gradient descent method to iterate each pixel, starting from the first row to the last row, as shown below: ; ; Represents the learning rate. Adjusting its value can be used to control the convergence speed. Represents the gradient direction of the objective function with respect to D; Indicates the error of the correction parameter D to the vertical stripes in the i-th row of pixels in image C; Indicates The calibration parameters for the iterations, No. The calibration parameters of the iterations are: .

[0040] Step 2: De-noising when hardware analog signal is taken: The infrared imaging system uses an infrared detector to receive infrared radiation emitted by an object and converts it into an electrical signal, which is then converted into a visible image through a signal processing circuit. In this process, when the analog signal is obtained from the infrared detector (sensor), it is usually interfered by various noises. These noises may come from the sensor itself, the circuit board, the environment and other factors. If effective denoising is not performed, the noise may affect the effect of subsequent image processing and cause image quality degradation. Therefore, when taking values ​​at the hardware level, the analog signal must be denoised to improve the signal-to-noise ratio of the image and ensure the accuracy of the signal. Median filtering is a classic digital signal processing method that is often used to eliminate impulse noise (also known as salt and pepper noise). The principle of the median filter is to select a sliding window, sort the signals in the window by size and take the median, thereby removing extreme noise points while retaining the edges and details of the signal.

[0041] Given a signal sequence, use a window, sort the values ​​in the window by size, select the middle value (i.e. the median after sorting) as the filter output, and slide the window until the entire signal sequence is traversed.

[0042] Step 3: short exposure to obtain a first infrared image; Capture the first infrared image by setting a short exposure time. Short exposure helps avoid overexposure of high brightness areas while capturing details in the image. The exposure time for short exposure can be set to 1 millisecond.

[0043] According to the initial lighting conditions of the scene and the characteristics of the sensor, a shorter exposure time value is determined. This value is determined by the preset parameters of the experiment to ensure that saturation does not occur in the high-brightness area and sufficient detail information can be captured. Using the set short exposure time parameters, real-time acquisition of infrared images is performed. At this point, the brightness distribution in the image mainly reflects the details in the high-brightness area, while the darker areas may not be obvious. The purpose of this step is to maximize the retention of high-frequency details in the image and avoid the loss of details in the bright area.

[0044] Step 4: Fourier transform to retain high frequencies; In this step, the first infrared image obtained from step 3 is subjected to Fourier transform to convert the image from the spatial domain to the frequency domain. In the frequency domain, the high-frequency components of the image mainly contain the edge and detail information of the image. By extracting and retaining these high-frequency components, the detail expression of the image can be enhanced, providing a basis for subsequent image processing steps.

[0045] Fourier transform is a mathematical transformation that converts a signal from the time or space domain to the frequency domain. For a two-dimensional image f(x,y), its discrete Fourier transform (Discrete Fourier Transform, DFT) can be expressed as: ; Where: f(x,y) is the original image in the spatial domain, F(u,v) is the Fourier coefficient in the frequency domain; M and N are the width and height of the image respectively; u and v are the coordinates in the frequency domain; The frequency index u is normalized to be in the range of [0,1]. can be considered as a phase change corresponding to a periodic signal; The frequency index v is normalized to be in the range [0,1]. can be considered as a phase change corresponding to a periodic signal; Describes the phase rotation of the signal at each point (x, y) in the spatial domain corresponding to a specific frequency (u, v) in the frequency domain; The Inverse Discrete Fourier Transform (IDFT) is used to convert the frequency domain data back to the spatial domain: ; The first infrared image is input into the Fourier transform calculation module to obtain the frequency domain representation F(u,v) of the image. At this point, the frequency information of the image has been decomposed into low-frequency and high-frequency components. The low-frequency component mainly contains the overall brightness and large-scale structural information of the image, while the high-frequency component contains the edge, texture and other detailed information of the image.

[0046] In the frequency domain image F(u,v), the part close to the center of the frequency domain is the low-frequency component, and the part closer to the edge is the high-frequency component. In order to enhance the details of the image, we need to retain the high-frequency component. The specific operation is: Set a frequency threshold Threshold, set the frequency components below the threshold to zero, and only retain the frequency components above the frequency threshold. The frequency threshold Threshold = (M + N) * 0.2, where M and N are the width and height of the image respectively. Form a new frequency domain image ,in: ; For the frequency domain image that retains the high frequency components Perform an inverse Fourier transform (IDFT) to convert it back to the spatial domain to obtain a high-frequency detail image This image mainly shows edges and details, with less overall brightness information. The high-frequency components of the first infrared image are extracted and enhanced, providing the necessary basis for subsequent image fusion and overall detail enhancement.

[0047] Step 5: reset the exposure value according to the judgment index to obtain a second infrared image; Compared with step 3, this exposure time is different to capture details of different brightness areas. According to the preset image quality indicator "global variance of the image", the details and noise level of the current first infrared image are judged. The image quality indicator "global variance of the image" can judge the details and noise level of the image. Calculate the global variance of the image, adjust the exposure parameters according to the size of the global variance, collect again, and obtain the second infrared image. The specific formula for calculating the global variance of the image is as follows: ; Where s is the global variance of the first infrared image; f(x+1,y) is the value of the (x+1,y) pixel of the first infrared image, where f(x,y) is the value of the (x,y) pixel of the first infrared image, where f(x,y+1) is the value of the (x,y+1) pixel of the first infrared image.

[0048] The new exposure parameter EX (in milliseconds) is calculated as follows: ; ; Where ra is a new exposure parameter calculated from the global variance of the over-exposure image and the global variance of the short-exposure image. In this embodiment, The value is 1108869, which is a specific value for our company's instrument after experiments.

[0049] Step 6: Fourier transform to retain low frequencies; Perform Fourier transform on the second infrared image to extract the low-frequency components of the image. The low-frequency components mainly reflect the overall brightness and structural information of the image. This step is similar to Step 4. Set a frequency threshold Threshold, set the frequency components above the threshold to zero, and only keep the frequency components below the threshold. Form a new frequency domain image .

[0050] Step 7: Fusing the high-frequency portion of the first infrared image obtained in step 4 and the low-frequency portion of the second infrared image obtained in step 6.

[0051] This step fuses the high-frequency components of the first infrared image with the low-frequency components of the second infrared image to generate an infrared image with rich details and uniform brightness. The fused image retains the details of the highlight area and fully displays the structure and brightness information of the overall image.

[0052] Use linear weighted fusion. is the high frequency component of the first infrared image, is the low-frequency component of the second infrared image, and the fused image It is expressed as: ; Wherein, α and β are fusion weight coefficients, satisfying α+β=1. In order to pay more attention to the details of the image, a larger α value is selected. As a specific embodiment, the value of α can be specifically set to 0.6.

[0053] Step 8: Digital image detail enhancement; After the image fusion is completed, in order to further improve the clarity and detail expression of the image, nonlinear detail enhancement techniques such as adaptive histogram equalization (CLAHE) and logarithmic transformation are used to process the fused image. Adaptive histogram equalization helps to enhance the local contrast of the image, while logarithmic transformation can effectively enhance the details of the low-brightness area in the image. The combination of these two methods can significantly improve the visual quality of the final image.

[0054] Color adaptive histogram equalization (CLAHE) is a technique for histogram equalization based on local areas (tiles). It can avoid the noise amplification and over-enhancement problems caused by traditional global histogram equalization and is particularly suitable for detail enhancement.

[0055] The specific steps are as follows: 1. The fused image It is divided into several small sub-areas, called tiles. Each tile is usually 16×16 pixels in size.

[0056] 2. Calculate the histogram for each tile separately and perform histogram equalization on it to make the grayscale distribution in the area more uniform. The equalized image will have better local contrast and highlight the details.

[0057] The specific steps of histogram equalization are as follows: 1. Calculate the grayscale histogram of the original image: count the number of times each gray level appears in the image.

[0058] 2. Normalized histogram: Convert the number of occurrences of each gray level into probability, that is, the frequency of each gray level.

[0059] 3. Calculate the cumulative distribution function (CDF): CDF is the accumulation of probability histograms, which represents the proportion of pixels less than or equal to a certain gray level.

[0060] 4. Mapping grayscale values: Use CDF to remap the grayscale values ​​of the original image. The mapping formula is:

[0061] Among them, g is the gray value after equalization, f is the original gray value, and L is the gray level (4096 in our instrument) 3. During the histogram equalization process of each tile, contrast limiting is applied to avoid noise amplification due to over-equalization. Contrast limiting is usually implemented by setting a threshold, and the histogram cumulative values ​​exceeding this threshold will be clipped and evenly distributed to the rest of the histogram.

[0062] 4. After processing each tile, use bilinear interpolation to smooth the boundaries between adjacent tiles to avoid block effects, thereby ensuring the smoothness and consistency of the overall image. The final image is .

[0063] Logarithmic transformation is a commonly used nonlinear enhancement technology, which is mainly used to enhance the details of low-brightness areas in images and increase the contrast of these areas. Since logarithmic transformation can compress the brightness range, it has a significant effect on enhancing the dark details in images.

[0064] Adaptive histogram equalization of the image After logarithmic transformation, the transformed image g(x,y) can be expressed as: ; Among them, c is a constant used to adjust the amplitude of the logarithmic transformation, and its value is usually: ; Where L is the grayscale level of the image (for a 12-bit image, L=4096), and M is the maximum grayscale value of the original image. The grayscale value range of the image after logarithmic transformation is usually small. In order to make the image display better, the grayscale of the transformed image can be stretched to map its grayscale value to the entire display range. .

[0065] In the process of detail enhancement, noise may be introduced or amplified. Therefore, in the last step, a bilateral filter or other smoothing technique is used to control the noise of the image, retaining the details while suppressing unnecessary noise.

[0066] The solution of the present application greatly improves the adaptability to complex noise scenes, enhances the adaptability of stripe removal, and realizes a more efficient infrared image processing capability.

[0067] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0068] In addition, the method of the present invention is not limited to being executed in the time sequence described in the specification, and may also be executed in other time sequences, in parallel or independently. Therefore, the execution order of the method described in this specification does not limit the technical scope of the present invention.

[0069] Although the present invention has been disclosed above by describing specific embodiments of the present invention, it should be understood that all the above embodiments and examples are exemplary rather than restrictive. Those skilled in the art may design various modifications, improvements or equivalents of the present invention within the spirit and scope of the appended claims. These modifications, improvements or equivalents should also be considered to be included in the protection scope of the present invention.

Claims

1. A method for enhancing the details of an infrared imaging system by combining FPGA, characterized in that: include: Step 1: Perform stripe non-uniformity correction on the infrared image at the FPGA level; Step 2: De-noising when taking the hardware analog signal value; Step 3: short exposure to obtain a first infrared image; Step 4, performing Fourier transform on the first infrared image obtained in step 3, retaining high frequencies; Step 5: reset the exposure value according to the judgment index to obtain a second infrared image; Step 6, performing Fourier transform on the second infrared image obtained in step 5, retaining the low frequency; Step 7, fusing the high-frequency portion of the first infrared image obtained in step 4 and the low-frequency portion of the second infrared image obtained in step 6; Step 8, enhancing the details of the digital image fused in step 7; Wherein, in the step 1, stripe non-uniformity correction is performed at the FPGA level, specifically including: The infrared image with stripe non-uniform noise is expressed as: ； In the above formula, represents pixel coordinates, and represent the image with vertical stripes and the ideal image respectively, represents vertical stripe noise; The above infrared image is subjected to fringe non-uniformity correction. The correction algorithm steps are as follows: Step 1: Rearrange the pixel values ​​of each column in the infrared image so that the pixel values ​​of each column are arranged in the smallest, largest, second smallest, and second largest order. The image obtained by this arrangement is recorded as C. Step 2: Optimize parameters by minimizing: ； Where i represents the row coordinate of the pixel coordinate, D and E represent the correction parameter and constant respectively; Represents the pixel in the i-th row of image C; is the objective function, which represents the error of the correction parameter D to the vertical stripes; Step 3: Use the gradient descent method to iterate each pixel, starting from the first row to the last row, as shown in the following formula: ； ； Represents the learning rate. Adjusting its value can be used to control the convergence speed. Represents the gradient direction of the objective function with respect to D; Indicates the error of the correction parameter D to the vertical stripes in the i-th row of pixels in image C; Indicates +1 iteration of calibration parameters, No. The calibration parameters of the iterations are: ; An represents an image containing vertical stripes; the number of rows of the matrix is ​​the number of rows of the image An.

2. The method for enhancing the details of an infrared imaging system by combining FPGA according to claim 1, characterized in that: In step 2, median filtering is used to remove noise when the hardware analog signal is taken: given a signal sequence, a window is used, the values ​​in the window are sorted by size, the middle value is selected as the filter output, and the window is slid until the entire signal sequence is traversed.

3. The method for enhancing details of an infrared imaging system using a combined FPGA according to claim 1, characterized in that: In step 4, Fourier transform is performed on the first infrared image obtained in step 3, which specifically includes: Performing Fourier transformation on the first infrared image to convert the image from the spatial domain to the frequency domain; For a two-dimensional image f(x,y), its discrete Fourier transform is expressed as: ； Where: f(x,y) is the original image in the spatial domain, F(u,v) is the Fourier coefficient in the frequency domain; M and N are the width and height of the image respectively; u and v are the coordinates in the frequency domain; The frequency index u is normalized to be in the range of [0,1]. can be considered as a phase change corresponding to a periodic signal; The frequency index v is normalized to be in the range [0,1]. can be considered as a phase change corresponding to a periodic signal; Describes the phase rotation of the signal at each point (x, y) in the spatial domain corresponding to a specific frequency (u, v) in the frequency domain; The inverse Fourier transform is used to convert the frequency domain data back to the spatial domain: ； The first infrared image is input into the Fourier transform calculation module to obtain the frequency domain representation F(u,v) of the image and retain its high frequency components. The specific operations are as follows: Set a frequency threshold, set the frequency components below the threshold to zero, and only retain the frequency components above the frequency threshold, forming a new frequency domain image that retains the high-frequency components. ,in: ； For the frequency domain image that retains the high frequency components Perform an inverse Fourier transform to convert it back to the spatial domain to obtain a high-frequency detail image .

4. The method for enhancing details of an infrared imaging system by combining FPGA according to claim 1, characterized in that: Step 5: resetting the exposure value according to the judgment index, specifically including: taking the global variance of the image as the image quality index, judging the details and noise level of the current first infrared image according to the preset image quality index; calculating the global variance of the image, and adjusting the exposure parameters according to the size of the global variance, and acquiring the second infrared image again; Among them, the specific formula for calculating the global variance of the image is as follows: ； Where s is the global variance of the first infrared image; f(x+1,y) is the value of the (x+1,y) pixel of the first infrared image, where f(x,y) is the value of the (x,y) pixel of the first infrared image, where f(x,y+1) is the value of the (x,y+1) pixel of the first infrared image; N x is the number of pixels of the image on the x-axis, N y is the number of pixels of the image on the y-axis; The new exposure parameter EX is calculated as follows: ； ； Where ra is the new exposure parameter calculated from the global variance of the over-exposure image and the global variance of the short-exposure image, and s t It is the average value of the global variance of multiple over-exposure images collected by the instrument. The unit of EX is milliseconds.

5. The method for enhancing details of an infrared imaging system by combining FPGA according to claim 3, characterized in that: Step 7: Fusing the high-frequency portion of the first infrared image obtained in step 4 with the low-frequency portion of the second infrared image obtained in step 6, specifically: is the high frequency component of the first infrared image, is the low-frequency component of the second infrared image, and the fused image It is expressed as: ； in, and is the fusion weight coefficient, satisfying α+β=1.

6. The method for enhancing details of an infrared imaging system by combining FPGA according to claim 1, characterized in that: Step 8: Digital image detail enhancement, specifically, using adaptive histogram equalization and logarithmic transformation to process the fused image in sequence.

7. The method for enhancing details of an infrared imaging system by combining FPGA according to claim 6, characterized in that: The specific steps of adaptive histogram equalization are as follows: The fused image Divide into several small sub-areas, called tiles, each tile is usually 16×16 pixels in size; Calculate the histogram for each tile separately and perform histogram equalization on it to make the grayscale distribution in the area more uniform; During the histogram equalization process of each tile, contrast limiting is applied to avoid noise amplification due to over-equalization; After processing each tile, the bilinear interpolation method is used to smooth the boundaries between adjacent tiles to avoid block effects, thereby ensuring the smoothness and consistency of the overall image, and finally obtaining the image .

8. The method for enhancing details of an infrared imaging system by combining FPGA according to claim 7, characterized in that: Adaptive histogram equalization of the image After logarithmic transformation, the transformed image g(x,y) is expressed as: ； Among them, c is a constant used to adjust the amplitude of the logarithmic transformation, and its value is: ； Where L is the grayscale level of the image, and M is the maximum grayscale value of the original image.

9. A device for enhancing the details of an infrared imaging system in combination with FPGA, characterized in that: include: The non-uniformity correction module is used to perform stripe non-uniformity correction on infrared images at the FPGA level; De-noising module, used to remove noise when hardware analog signal is taken; A first infrared image acquisition module, used to obtain a first infrared image through short exposure; A high-frequency acquisition module, used for performing Fourier transform on the obtained first infrared image to retain the high-frequency part; A second infrared image acquisition module, used to reset the exposure value according to the judgment index to obtain a second infrared image; A low-frequency acquisition module, used for performing Fourier transform on the second infrared image and retaining the low-frequency part; A fusion module, used for fusing the high frequency part of the first infrared image with the low frequency part of the second infrared image; A detail enhancement module is used to enhance the details of the fused digital image; Among them, in the non-uniformity correction module, stripe non-uniformity correction is performed at the FPGA level, specifically including: The infrared image with stripe non-uniform noise is expressed as: ； In the above formula, represents pixel coordinates, and represent the image with vertical stripes and the ideal image respectively, represents vertical stripe noise; The above infrared image is subjected to fringe non-uniformity correction. The correction algorithm steps are as follows: Step 1: Rearrange the pixel values ​​of each column in the infrared image so that the pixel values ​​of each column are arranged in the smallest, largest, second smallest, and second largest order. The image obtained by this arrangement is recorded as C. Step 2: Optimize parameters by minimizing: ； Where i represents the row coordinate of the pixel coordinate, D and E represent the correction parameter and constant respectively; Represents the pixel in the i-th row of image C; is the objective function, which represents the error of the correction parameter D to the vertical stripes; Step 3: Use the gradient descent method to iterate each pixel, starting from the first row to the last row, as shown below: ； ； Represents the learning rate. Adjusting its value can be used to control the convergence speed. Represents the gradient direction of the objective function with respect to D; Indicates the error of the correction parameter D to the vertical stripes in the i-th row of pixels in image C; Indicates +1 iteration of calibration parameters, No. The calibration parameters of the iterations are: ; An represents an image containing vertical stripes; the number of rows of the matrix is ​​the number of rows of the image An.

Images