FPGA-based multi-scale fusion image enhancement method

Abstract

The application discloses a multi-scale fusion image enhancement method based on FPGA, taking an infrared image as an example, and comprises the following steps: controlling an infrared detector to output analog data, and obtaining a 14 / 16 bit corrected image after the analog data pass through a front-end circuit and pre-processing FPGA software; performing median filtering on the corrected image to reduce singular points; then filtering the filtered image using different sizes of Gaussian kernels respectively, and sequentially performing difference to obtain edge information under different scales; and finally, superimposing the edge information on the median filtered image according to different weights to obtain an enhanced infrared image. The application has high real-time performance and strong universality, fully considers the feature extraction problem of different scales, and has good enhancement effect in multiple scenes.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a multi-scale fusion image enhancement method based on FPGA. Background Technology

[0002] In the field of image processing, especially infrared image processing, traditional image enhancement methods mainly include histogram equalization, contrast stretching, and local contrast enhancement. These methods can improve image quality to some extent, but because they typically rely excessively on global statistical properties, they cannot effectively handle dynamically changing scenes, such as target motion and illumination changes. Furthermore, traditional methods face problems of high computational complexity and poor real-time performance when processing high-resolution, large-size infrared images.

[0003] Due to its strong parallel processing capabilities and high flexibility, FPGA has gradually become a commonly used platform for high-speed, real-time image processing. Currently, commonly used image enhancement methods typically use global information, requiring the storage of complete frame images before calculation during image processing. These algorithms are prone to prolonged image output time and excessive resource consumption. Therefore, this invention proposes an image enhancement algorithm based on serial data processing. This method has strong real-time performance and can further reduce the consumption of computing resources through methods such as lookup tables. Summary of the Invention

[0004] The purpose of this invention is to propose an infrared image and video enhancement method that overcomes the shortcomings of existing technologies, such as poor real-time performance, significant noise enhancement, and weak image enhancement effects. This method can improve image quality, enhance visual effects, and has good versatility and real-time performance. In this regard, embodiments of the present invention also provide an infrared image and video enhancement method for FPGA, comprising: The infrared imaging detector is controlled by the front-end circuit and pre-processing FPGA software, and the analog signal output by the detector is converted into 14 / 16-bit raw infrared image data. After non-uniform correction processing, the infrared video image to be enhanced is obtained. Median filtering is applied to the calibration graph to reduce singularities; The filtered image is then processed through a feature pyramid to obtain feature information at different scales (the image is filtered using Gaussian kernels of different scales). (1) (2) (3) in, It is the standard deviation, which controls the smoothness of the filter (the larger the value, the more obvious the blurring effect). It refers to the position of a pixel within the kernel relative to the center; For kernel coordinates, The radius of the core; coefficient Used for normalization to ensure that the sum of the Gaussian kernels is 1.

[0005] Using FPGAs, lookup tables can be used to store pre-calculated results, avoiding complex mathematical operations at runtime, reducing computation time, and improving real-time performance. Image reconstruction is performed by successively subtracting feature information at different scales and then applying different weights. The mathematical expression for this is: (4) in Indicates the first Layer feature maps, especially .

[0006] The beneficial effects of this invention are as follows: Due to the advantages of FPGAs, such as strong parallel processing capabilities and high flexibility, this invention proposes an FPGA-based image enhancement algorithm. The algorithm uses a sliding window to filter the image and obtain feature images at different scales, i.e., the high-frequency features of the image. By assigning different weights to the feature images at different scales, image enhancement is achieved. This method has low data latency, strong real-time performance, and can save a lot of cache space and computing resources. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating the present invention. Detailed Implementation

[0008] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this document. Specific Implementation Example 1: The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0010] To enable those skilled in the art to better understand the technical solution, the following describes in detail, with reference to the accompanying drawings, a multi-scale fusion image enhancement algorithm based on FPGA provided by the present invention.

[0011] like Figure 1As shown, this embodiment of the invention provides a multi-scale fusion image enhancement algorithm based on FPGA, including: Step S1: After obtaining the analog signal of the infrared detector through the A / D chip and digitizing the signal, the data is input into the preprocessing FPGA software to sort and reassemble the data to obtain the original infrared image. In order to compensate for the non-uniformity of the infrared detector imaging, the original image needs to be non-uniformly corrected to obtain an infrared image with less noise. Step S2, median filtering of the calibration image: Since there may be some independent singularities and noise in the infrared detector imaging feature image, median filtering is required to avoid abnormal amplification of noise in subsequent image enhancement. In this embodiment, the specific method is as follows: use The window performs sliding filtering on the image, sorts all pixel values ​​in the window, and selects the median value to replace the grayscale value at the current coordinate.

[0012] Step S3: The median-filtered image is used as the input image and processed through a feature pyramid to obtain feature information at different scales. Taking a three-layer feature pyramid as an example, this embodiment is specifically as follows: To obtain image edge information at different scales, the median-filtered image first needs to be convolved with three different scale kernels to obtain feature images at different scales. The mathematical expression for convolution operation is defined as follows: (1) in For the convolution kernel, each pixel The new value is a weighted sum of neighboring pixels, with the weights determined by the Gaussian kernel; image boundary processing is completed after padding. The mathematical expression of the two-dimensional Gaussian function is defined as follows: (2) In this example, to reduce computational complexity, convolution operations need to be discretized in practical applications, i.e., based on the kernel size, such as ( Generate discrete weights and renormalize them: (3) in For kernel coordinates, For the kernel radius (e.g.) convolution kernel );coefficient Used for normalization to ensure that the sum of the Gaussian kernels is 1.

[0013] To further reduce the FPGA resource consumption of convolution operations, a lookup table can be used to pre-generate the convolution kernel.

[0014] Step S4: By setting k to 3, 5, and 7 respectively, feature images obtained from three different scale convolution operations in S3 are obtained. These feature images are then subtracted in descending order of convolution kernel size. The feature image obtained from the smallest convolution kernel is subtracted from the median-filtered image to obtain three edge information points. These edge information points are then fused with the remaining original images using different weights to obtain the enhanced infrared image. Its mathematical expression is defined as follows: (4) in Indicates the first Layer feature maps, especially The image at layer 0 is the original input image.

[0015] This invention discloses a highly real-time and versatile infrared image data enhancement method based on an FPGA platform. Utilizing the parallel processing capabilities of FPGA, it can process serially input image data in real time. By performing convolution operations through a sliding window method, the high-frequency components of the image are obtained. This method avoids the large storage space consumption caused by the need for whole-frame caching in image enhancement. It can process images of different resolutions and effectively shortens the image latency, ensuring the real-time performance of image data. It has been tested in multiple scenarios and has shown good enhancement effects. Specific Implementation Example 2: A multi-scale fusion image enhancement method based on FPGA, the method comprising: The infrared image after median filtering is processed by convolution at different scales to obtain feature images at different scales. Each feature image at each scale is then processed by two-dimensional Gaussian to reduce computational complexity. Discrete weights are generated based on the size of the convolution kernel during the convolution process, and the discrete weights are normalized to obtain the weights corresponding to each convolution kernel. Multiple feature images are subtracted sequentially according to their convolution kernel size to obtain edge information of the feature image differences. The feature image obtained by the smallest convolution kernel is subtracted from the median-filtered infrared image to obtain edge information of the infrared image. Both the edge information of the feature image differences and the edge information of the infrared image are edge information. The edge information is fused with the corresponding median-filtered infrared image with different weights to obtain an enhanced infrared image. The enhanced infrared image is used for time-axis correlation processing in video image processing on the FPGA platform.

[0017] Furthermore, before performing convolution processing at different scales on the median-filtered infrared image to obtain feature images at different scales, the method further includes: The infrared signal from the infrared detector is acquired, and the raw infrared image is obtained through FPGA preprocessing. Non-uniform correction is then applied to the raw infrared image to obtain an infrared image with less noise. The corrected infrared image is then subjected to median filtering to obtain the median-filtered infrared image.

[0018] Furthermore, the infrared image after median filtering is processed by convolution at different scales to obtain feature images at different scales. Each feature image at each scale undergoes two-dimensional Gaussian processing to reduce computational complexity, including: To obtain image edge information at different scales, the median-filtered image first needs to be convolved with three different scale convolution kernels to obtain feature images at different scales. The mathematical expression for convolution operation is defined as follows: (1) in, For convolution kernel, The kernel radius is the radius of the convolution kernel. For pixel values, The pixel values ​​after convolution processing, each pixel The new value is the weighted sum of the neighboring pixels; Image boundary processing is performed by padding, and the mathematical expression of the two-dimensional Gaussian function is defined as follows: (2) Among them, is, For pixel coordinates, Standard deviation, It is a two-dimensional Gaussian function.

[0019] Furthermore, the step of generating discrete weights based on the size of the convolution kernel during convolution processing, and then normalizing these discrete weights to obtain the weights corresponding to each convolution kernel, includes: Discrete weights are generated based on the size of the convolution kernel and then renormalized: (3) in For kernel coordinates, For kernel radius; coefficient Used for normalization, Standard deviation, The Gaussian value of the kernel coordinates. The result is a normalization of the discrete weights, ensuring that the sum of the Gaussian kernels is 1.

[0020] Further, the process involves sequentially subtracting multiple feature images according to their convolution kernel size to obtain edge information of the feature image differences; subtracting the feature map obtained from the smallest convolution kernel from the median-filtered infrared image to obtain edge information of the infrared image; both the edge information of the feature image differences and the edge information of the infrared image are considered edge information; and fusing the edge information with the corresponding median-filtered infrared image using different weights to obtain the enhanced infrared image, including: The feature images obtained by convolution operations at different scales are sorted in order of convolution kernel size; the sorted adjacent feature images are then subtracted to obtain the first and second edge information of the feature image difference; The third edge information is obtained by subtracting the feature image corresponding to the smallest convolution kernel from the infrared image after median filtering. The first edge information, the second edge information, and the third edge information are fused with the median-filtered infrared image with different weights to obtain the enhanced infrared image; The kernel radii of the convolution kernels are set to 3, 5, and 7 respectively, and their mathematical expressions are as follows: (4) in, For the first The weights assigned to layer feature maps during image enhancement. For the final enhanced image output, Indicates the first Layer feature map.

[0021] A multi-scale fusion image enhancement system based on FPGA, the system comprising: a convolution processing unit, a normalization unit, and a correlation unit; The convolution processing unit is used to perform convolution processing on the median-filtered infrared image at different scales to obtain feature images at different scales. Each feature image at a scale is processed by two-dimensional Gaussian to reduce the computational complexity. The normalization unit is used to generate discrete weights based on the size of the convolution kernel during the convolution process, and then normalize the discrete weights to obtain the weights corresponding to each convolution kernel. The correlation unit is used to perform subtraction processing on multiple feature images sequentially according to the size of the convolution kernel to obtain the edge information of the feature image difference; the feature map obtained by the smallest convolution kernel is subtracted from the median-filtered infrared image to obtain the edge information of the infrared image; the edge information of the feature image difference and the edge information of the infrared image are both edge information; the edge information is fused with the corresponding median-filtered infrared image with different weights to obtain the enhanced infrared image, which is used for correlation processing on the time axis in the video image processing of the FPGA platform.

[0022] Preferably, the system further includes a preprocessing unit, the preprocessing unit being used for: The infrared signal from the infrared detector is acquired and preprocessed by the FPGA to obtain the original infrared image; the original infrared signal and the retrograde signal are non-uniformly corrected to obtain an infrared image with less noise. The corrected infrared image is then subjected to median filtering to obtain the median-filtered infrared image.

[0023] Preferably, the convolution processing unit is used for: To obtain image edge information at different scales, the median-filtered image first needs to be convolved with three different scale kernels to obtain feature images at different scales. The mathematical expression for convolution operation is defined as follows: in, For convolution kernel, The kernel radius is the radius of the convolution kernel. For pixel values, The pixel values ​​after convolution processing, each pixel The new value is the weighted sum of the neighboring pixels; Image boundary processing is performed by padding, and the mathematical expression of the two-dimensional Gaussian function is defined as follows: Among them, is, For pixel coordinates, Standard deviation, It is a two-dimensional Gaussian function.

[0024] Preferably, the normalization unit is used for: Discrete weights are generated based on the size of the convolution kernel and then renormalized: in For kernel coordinates, For kernel radius; coefficient Used for normalization, Standard deviation, The Gaussian value of the kernel coordinates. The result is a normalization of the discrete weights, ensuring that the sum of the Gaussian kernels is 1.

[0025] Preferably, the associative unit is used for: The feature images obtained by convolution operations at different scales are sorted in order of convolution kernel size; the sorted adjacent feature images are then subtracted to obtain the first and second edge information of the feature image difference; The third edge information is obtained by subtracting the feature image corresponding to the smallest convolution kernel from the infrared image after median filtering. The first edge information, the second edge information, and the third edge information are fused with the median-filtered infrared image with different weights to obtain the enhanced infrared image; The kernel radii of the convolution kernels are set to 3, 5, and 7 respectively, and their mathematical expressions are as follows: in, For the first The weights assigned to layer feature maps during image enhancement. For the final enhanced image output, Indicates the first Layer feature map.

[0026] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some embodiments, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth by the appended claims.

Claims

1. A multi-scale fusion image enhancement method based on FPGA, characterized in that, The method includes: The infrared image after median filtering is processed by convolution at different scales to obtain feature images at different scales. Each feature image at each scale is then processed by two-dimensional Gaussian to reduce computational complexity. Discrete weights are generated based on the size of the convolution kernel during the convolution process, and the discrete weights are normalized to obtain the weights corresponding to each convolution kernel. Multiple feature images are subtracted sequentially according to their convolution kernel size to obtain edge information of the feature image differences. The feature image obtained by the smallest convolution kernel is subtracted from the median-filtered infrared image to obtain edge information of the infrared image. Both the edge information of the feature image differences and the edge information of the infrared image are edge information. The edge information is fused with the corresponding median-filtered infrared image with different weights to obtain an enhanced infrared image. The enhanced infrared image is used for time-axis correlation processing in video image processing on the FPGA platform.

2. The FPGA-based multi-scale fusion image enhancement method as described in claim 1, characterized in that, Before performing convolution processing at different scales on the median-filtered infrared image to obtain feature images at different scales, the method further includes: The infrared signal from the infrared detector is acquired, and the raw infrared image is obtained through FPGA preprocessing. Non-uniform correction is then applied to the raw infrared image to obtain an infrared image with less noise. The corrected infrared image is then subjected to median filtering to obtain the median-filtered infrared image.

3. The FPGA-based multi-scale fusion image enhancement method as described in claim 1, characterized in that, The process involves performing convolutional processing on the median-filtered infrared image at different scales to obtain feature images at different scales. Each feature image at a different scale undergoes two-dimensional Gaussian processing to reduce computational complexity, including: To obtain image edge information at different scales, the median-filtered image first needs to be convolved with three different scale convolution kernels to obtain feature images at different scales. The mathematical expression for convolution operation is defined as follows: （1） in, For convolution kernel, The kernel radius is the radius of the convolution kernel. For pixel values, The pixel values ​​after convolution processing, each pixel The new value is the weighted sum of the neighboring pixels; Image boundary processing is performed by padding, and the mathematical expression of the two-dimensional Gaussian function is defined as follows: （2） Among them, is, For pixel coordinates, Standard deviation, It is a two-dimensional Gaussian function.

4. The FPGA-based multi-scale fusion image enhancement method as described in claim 1, characterized in that, The process of generating discrete weights based on the size of the convolution kernel during convolution processing, and then normalizing these discrete weights to obtain the weights corresponding to each convolution kernel, includes: Discrete weights are generated based on the size of the convolution kernel and then renormalized: （3） in For kernel coordinates, For kernel radius; coefficient Used for normalization, Standard deviation, The Gaussian value of the kernel coordinates. The result is a normalization of the discrete weights, ensuring that the sum of the Gaussian kernels is 1.

5. The FPGA-based multi-scale fusion image enhancement method as described in claim 4, characterized in that, The feature images are subtracted sequentially according to the size of the convolution kernel to obtain the edge information of the feature image difference; the feature image obtained by the smallest convolution kernel is subtracted from the infrared image after median filtering to obtain the edge information of the infrared image; the edge information of the feature image difference and the edge information of the infrared image are both edge information. Edge information is fused with the corresponding median-filtered infrared image using different weights to obtain an enhanced infrared image, including: The feature images obtained by convolution operations at different scales are sorted in order of convolution kernel size; the sorted adjacent feature images are then subtracted to obtain the first and second edge information of the feature image difference; The third edge information is obtained by subtracting the feature image corresponding to the smallest convolution kernel from the infrared image after median filtering. The first edge information, the second edge information, and the third edge information are fused with the median-filtered infrared image with different weights to obtain the enhanced infrared image; The kernel radii of the convolution kernels are set to 3, 5, and 7 respectively, and their mathematical expressions are as follows: （4） in, For the first The weights assigned to layer feature maps during image enhancement. For the final enhanced image output, Indicates the first Layer feature map.

6. A multi-scale fusion image enhancement system based on FPGA, characterized in that, The system includes: a convolution processing unit, a normalization unit, and a correlation unit; The convolution processing unit is used to perform convolution processing on the median-filtered infrared image at different scales to obtain feature images at different scales. Each feature image at a scale is processed by two-dimensional Gaussian to reduce the computational complexity. The normalization unit is used to generate discrete weights based on the size of the convolution kernel during the convolution process, and then normalize the discrete weights to obtain the weights corresponding to each convolution kernel. The correlation unit is used to perform subtraction processing on multiple feature images sequentially according to the size of the convolution kernel to obtain the edge information of the feature image difference; the feature map obtained by the smallest convolution kernel is subtracted from the median-filtered infrared image to obtain the edge information of the infrared image; the edge information of the feature image difference and the edge information of the infrared image are both edge information; the edge information is fused with the corresponding median-filtered infrared image with different weights to obtain the enhanced infrared image, which is used for correlation processing on the time axis in the video image processing of the FPGA platform.

7. The FPGA-based multi-scale fusion image enhancement system as described in claim 1, characterized in that, The system further includes a preprocessing unit, which is used for: The infrared signal from the infrared detector is acquired and preprocessed by the FPGA to obtain the original infrared image; the original infrared signal and the retrograde signal are non-uniformly corrected to obtain an infrared image with less noise. The corrected infrared image is then subjected to median filtering to obtain the median-filtered infrared image.

8. The FPGA-based multi-scale fusion image enhancement system as described in claim 1, characterized in that, The convolution processing unit is used for: To obtain image edge information at different scales, the median-filtered image first needs to be convolved with three different scale kernels to obtain feature images at different scales. The mathematical expression for convolution operation is defined as follows: in, For convolution kernel, The kernel radius is the radius of the convolution kernel. For pixel values, The pixel values ​​after convolution processing, each pixel The new value is the weighted sum of the neighboring pixels; Image boundary processing is performed by padding, and the mathematical expression of the two-dimensional Gaussian function is defined as follows: Among them, is, For pixel coordinates, Standard deviation, It is a two-dimensional Gaussian function.

9. The FPGA-based multi-scale fusion image enhancement system as described in claim 1, characterized in that, Normalization units are used for: Discrete weights are generated based on the size of the convolution kernel and then renormalized: in For kernel coordinates, For kernel radius; coefficient Used for normalization, Standard deviation, The Gaussian value of the kernel coordinates. The result is a normalization of the discrete weights, ensuring that the sum of the Gaussian kernels is 1.

10. The FPGA-based multi-scale fusion image enhancement system as described in claim 9, characterized in that, The associative unit is used for: The feature images obtained by convolution operations at different scales are sorted in order of convolution kernel size; the sorted adjacent feature images are then subtracted to obtain the first and second edge information of the feature image difference; The third edge information is obtained by subtracting the feature image corresponding to the smallest convolution kernel from the infrared image after median filtering. The first edge information, the second edge information, and the third edge information are fused with the median-filtered infrared image with different weights to obtain the enhanced infrared image; The kernel radii of the convolution kernels are set to 3, 5, and 7 respectively, and their mathematical expressions are as follows: in, For the first The weights assigned to layer feature maps during image enhancement. For the final enhanced image output, Indicates the first Layer feature map.

Images