Infrared image detail enhancement method based on video stream

Abstract

The invention discloses an infrared image detail enhancement method based on a video stream, and the method comprises the steps: superposing an infrared original image in the video stream to obtain an original image data matrix, carrying out the bilateral filtering and extreme value extraction of the original image data matrix, carrying out the smoothing processing through combining with a historical frame average parameter, and obtaining a parameter enhanced image; and inputting the original image data matrix and the original image data matrix after bilateral filtering into a detail enhancement module for enhancement, carrying out smoothing processing in combination with a historical frame average parameter after extreme value extraction to obtain a parameter enhanced image, and finally, carrying out linear superposition on the parameter enhanced image to obtain an enhanced infrared original image. According to the method, optimization design is carried out on the basis of a single-frame infrared image detail enhancement algorithm, and compared with a single-frame infrared image enhancement algorithm, the enhancement algorithm based on the video stream needs to pay attention to the input image difference between frames and the influence of the difference on the enhanced video stream under the algorithm framework of the enhancement algorithm based on the video stream.

Description

Technical Field

[0001] The invention belongs to the technical field of image recognition, and in particular relates to a method for enhancing infrared image details based on video stream. Background Art

[0002] For a single-frame image, detail enhancement only needs to consider the image parameters of this frame and find the optimal solution within the parameter range of this frame to achieve the best enhancement effect under the algorithm framework. However, in most practical applications, the enhancement algorithm is oriented to the video stream rather than the single-frame image.

[0003] Unlike a single-frame image, for a multi-frame image sequence in a video stream, the image signal input will fluctuate, and the signal fluctuation includes signal changes and noise disturbances.

[0004] This kind of image sequence input with changes will cause obvious "flickering" problems for the single-frame digital detail enhancement algorithm (DDE). This is because the enhancement algorithm itself amplifies the image signal of the enhanced input, and the video input with changes between previous and subsequent frames and noise will amplify its changes, thus visually forming the result of image flickering.

[0005] For enhancement algorithms based on video streams, it is necessary to solve the problem of enhancement parameter fluctuations caused by fluctuations in image data of frames before and after the input image. The phenomenon caused by this problem is that the enhanced video will have "flickering". Summary of the invention

[0006] In order to solve the above problems in the prior art, the present invention proposes a method for enhancing infrared image details based on video stream.

[0007] The technical solution adopted by the present invention to solve the technical problem is: a method for enhancing infrared image details based on video stream, comprising the following steps:

[0008] S1, input N frames of 14-bit infrared original image data in the video stream, N ≥ 4, the multi-frame superposition module superimposes the Nth frame, the N-1th frame, the N-2th frame, the N-3th frame, ... images, and accumulates multiple frames to obtain a 16-bit original image data matrix;

[0009] S2, extract the bilateral filtering parameters of the original image data matrix, extract the maximum and minimum values, and then perform smoothing processing combined with the historical frame average parameters to obtain a smoothed parameter enhanced image;

[0010] S3, taking the original image data matrix and the original image data matrix after bilateral filtering as input to perform detail enhancement, extracting the maximum value and combining it with the historical frame average parameter to perform smoothing processing to obtain a smoothed parameter enhanced image;

[0011] S4, finally, linearly superimposing the parameter enhanced images of step S2 and step S3 to complete the enhanced infrared original image with the image sequence number N.

[0012] Further, when N=4, the step S1 is to first define the input image data matrix as A N-3 , A N-2 , A N-1 , A N , through formula D N =A N-3 +A N-2 +A N-1 +A N Superimpose the four input data matrices and perform matrix addition to obtain the original image data matrix D N , the element data width in this matrix is ​​16 bits.

[0013] Further, the step S2 is to convert the original image data matrix D N As the input of the detail enhancement algorithm, bilateral filtering is performed, and the image information of different dynamic ranges in the image is separated using the following formula. After bilateral filtering of the original image data matrix DN, the base image D with a large dynamic range image is obtained. base :

[0014]

[0015] The bilateral filter s(ix, jy) = g(x, y) r(ix, jy) in the formula is a Gaussian low-pass filter kernel acting in the spatial domain. Grayscale difference filter kernel Where f(x, y) represents the grayscale value at the (x, y) coordinate.

[0016] Furthermore, in the steps S2 and S3, the unsharp mask image enhancement method is used to separate the small dynamic range detail image to obtain the detail image D detail (i, j) = D N (i, j)-D base (i, j); After obtaining the base image and detail image, parameters of each image are extracted and weighted by adding the historical frame average parameters. The above calculation steps finally obtain the grayscale interval of the base image as The grayscale range of the detail image is

[0017] Furthermore, assuming that the minimum and maximum values ​​of the base layer extracted from the image of this frame (frame number is N) are Then the grayscale interval used for grayscale mapping in this frame is in and The calculation formula is as follows:

[0018]

[0019] The P value in the formula represents the number of cached frames of the parameter value. The larger the value, the higher the proportion of the historical value, and relatively speaking, the smaller the proportion of the parameters of this frame to the enhanced interval cutoff value.

[0020] Furthermore, for the small dynamic range detail map extracted by the unsharp mask, its maximum value is Then the grayscale interval used for grayscale mapping in this frame is in and The calculation formula is shown in the following two formulas:

[0021]

[0022] The P value in the formula represents the number of cached frames of the maximum parameter of the detail layer.

[0023] Furthermore, in step S4, after parameter extraction and smoothing, the 16-bit image obtained by superimposing multiple frames is quantized to an 8-bit visible image by enhancing image details:

[0024] (1) Let the 16-bit base image and detail image be and The corresponding statistical gray interval extraction and smoothing can be obtained Then the mapping quantization first normalizes the grayscale value according to the interval, as shown in the following two formulas:

[0025]

[0026]

[0027] After normalization, we get

[0028] (2) After normalization, grayscale mapping is performed on the base image and detail image to enhance the dynamic information of the two layers. The grayscale mapping curves are shown in the following two equations:

[0029]

[0030] Among them, b and c in the above formula are adjustment parameters. For the linear mapping and gamma transformation of the large dynamic base image, the stretched grayscale area and the compressed grayscale area of ​​the mapping can be balanced. For the detail image, the dark details need to be highlighted, so the mapping of the low grayscale interval is mainly performed, and the formula f is used. detail (x) = xγ for gamma transformation;

[0031] (3) Assume that the images after normalization and calculation by the above mapping curve are Then the image is quantized to 8 bit width:

[0032]

[0033] In the above two equations, [] is the integer symbol. After quantization, we get 8-bit base image and detail image. and Finally, the base image and the detail image are linearly superimposed to form the output image of this algorithm. The calculation process is as follows:

[0034] The beneficial effects produced by the present invention are:

[0035] The present invention is based on the infrared detail enhancement framework of a single frame in a video stream and utilizes the idea of ​​multi-frame superposition to reduce the random noise of the current image frame, increase the image bit width, and improve the image signal-to-noise ratio. At the same time, the key parameters of the infrared image enhancement processing are smoothed, and the smoothed parameter results are obtained by comprehensive calculation using the extracted historical frame average parameters and the current parameters.

[0036] The invention smoothes the enhancement parameters extracted from the image sequence, making the enhanced image sequence more stable and reducing the flickering phenomenon. At the same time, the algorithm of the invention accumulates multiple frames of input image frames, enhancing the target signal while reducing the disturbance of random noise, further making the input image signal more stable and improving the signal-to-noise ratio.

[0037] Finally, the present invention can effectively alleviate the image signal and parameter mutation problem between image sequence frames, and optimize the flickering phenomenon of image sequence caused by this problem. The present invention makes image detail enhancement more suitable for practical application scenarios, and at the same time improves the image signal-to-noise ratio and the enhancement effect of processing video. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The figure is a flow chart of the enhancement method of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] In order to solve the problem of practical application of the above-mentioned detail enhancement algorithm under video stream input, the present invention proposes a parameter smoothing method of the previous and next frames for the infrared detail enhancement algorithm of the video stream to alleviate the parameter fluctuation problem of the previous and next frames, and finally forms a method for detail enhancement of infrared image sequence with stable output.

[0041] The overall process of the algorithm of the present invention is as follows Figure 1 As shown, as a specific embodiment, the input is 4 frames of 14-bit infrared original image data. Assuming that the actual input image sequence number is N, the multi-frame superposition module will simultaneously obtain the cached historical frames N-1, N-2, and N-3 for superposition of four frames. After superposition, a 16-bit original image data matrix is ​​obtained, and then bilateral filtering parameters are extracted. After extraction, smoothing is performed in combination with the average parameters of the historical frames. After the maximum value is extracted, the enhancement parameters are obtained, and the parameter enhancement and superimposed image are input into the detail enhancement module for enhancement and quantization. Linear superposition completes the enhancement of the input original image with the image sequence number N. The detailed steps are as follows.

[0042] S1, input N frames of 14-bit infrared original image data in the video stream, N ≥ 4, the multi-frame superposition module superimposes the Nth frame, the N-1th frame, the N-2th frame, the N-3th frame, ... images, and accumulates multiple frames to obtain a 16-bit original image data matrix.

[0043] First, assume that the input image data matrices are A N-3 , A N-2 , A N-1 , A N , through formula D N =A N-3 +A N-2 +A N-1 +A N Superposition, N = 4, the 4 input data matrices are added to obtain the original image data matrix D N , the element data width in this matrix is ​​16 bits.

[0044] S2, extract the bilateral filtering parameters of the original image data matrix, extract the maximum and minimum values, and then perform smoothing processing combined with the historical frame average parameters to obtain a smoothed parameter enhanced image.

[0045] (1) Next, the original image data matrix D N Bilateral filtering is performed as the input of the detail enhancement algorithm to separate image information of different dynamic ranges in the image. The calculation process is shown in the following formula:

[0046]

[0047] The s(ix, jy) in the formula is a bilateral filter, and the calculation formula is s(ix, j-yy)=g(x, y)·r(ix, jy).

[0048] The g(x, y) in the formula is the Gaussian low-pass filter kernel acting in the spatial domain, and its specific calculation is shown in the following formula:

[0049]

[0050] The r(ix, jy) in the formula is the grayscale domain difference filter kernel, and its specific generation process is shown in the following formula:

[0051]

[0052] f(x, y) in the formula represents the grayscale value at the (x, y) coordinates.

[0053] In the original image data matrix D N After bilateral filtering, the base image D with a large dynamic range is obtained. base .

[0054] S3, taking the original image data matrix and the original image data matrix after bilateral filtering as input for detail enhancement, extracting the maximum value and combining it with the historical frame average parameter for smoothing to obtain a smoothed parameter enhanced image.

[0055] (2) Based on step (1), the unsharp mask image enhancement method is used to separate the small dynamic range detail image to obtain the detail image D detail (i, j) = D N (i, j)-D base (i, j).

[0056] After obtaining the base image and detail image, parameters of each image are extracted and the historical frame average parameters are added for weighting.

[0057] Assume that the minimum and maximum values ​​of the base layer extracted from the image of this frame (frame number is N) are Then the grayscale interval used for grayscale mapping in this frame is in and The calculation formula is as follows:

[0058]

[0059] The P value represents the number of cached frames of the parameter value. The larger the value, the higher the proportion of the historical value, and relatively speaking, the smaller the proportion of the parameter of the current frame to the enhanced interval cutoff value. In this embodiment, the P value is 64.

[0060] (3) Similarly, for the small dynamic range detail image extracted by the unsharp mask, its maximum value is Then the grayscale interval used for grayscale mapping in this frame is in and The calculation formula is shown in the following two formulas:

[0061]

[0062] The P value represents the number of cache frames of the maximum value parameter of the detail layer. In this embodiment, the P value is 64.

[0063] The above calculation steps finally obtain the grayscale interval of the base image as follows: The grayscale range of the detail image is

[0064] S4, finally, linearly superimposing the parameter enhanced images of step S2 and step S3 to complete the enhanced infrared original image with the image sequence number N.

[0065] After the above-mentioned parameter extraction and smoothing processing, the subsequent image detail enhancement quantizes the 16-bit image obtained by superimposing multiple frames into an 8-bit visible image.

[0066] (1) Let the 16-bit base image and detail image be and The corresponding statistical grayscale interval can be obtained by the above extraction and smoothing method: Then the mapping quantization first normalizes the grayscale value according to the interval, as shown in the following two formulas:

[0067]

[0068] After normalization, we get

[0069] (2) After normalization, grayscale mapping is performed on the base image and detail image to enhance the dynamic information of the two layers. The grayscale mapping curves are shown in the following two equations:

[0070]

[0071] Among them, b and c in the above formula are adjustment parameters, and the algorithm of the present invention takes the parameters as 0.45 and 4 respectively. This formula can balance the stretched grayscale area and compressed grayscale area of ​​the mapping for the linear mapping and gamma transformation of the large dynamic base image. If the detail image needs to highlight the dark details, the mapping of the low grayscale interval is mainly performed, and the formula f is used detail (x) = x γ Perform a gamma transformation. The exponent used in the present invention is 1.5.

[0072] (3) Assume that the images after normalization and calculation by the above mapping curve are Then the image is quantized to 8 bit width:

[0073]

[0074] In the above two formulas, "[]" is a rounding symbol. After quantization, we get 8-bit base image and detail image. and Finally, the base image and the detail image are linearly superimposed to form the output image of this algorithm. The calculation process is as follows, where the weight p is 0.4:

[0075] The above steps complete the enhancement and quantization process of the image sequence from 14-bit images to 8-bit images. First, this algorithm uses a multi-frame accumulation method to improve the image signal-to-noise ratio. It is particularly important to note that the more superimposed frames there are, the better the effect. Because there are moving targets in the image, when the number of frames is increased to a certain number or the target speed in the field of view is greater than a certain threshold, the superimposed image will have obvious smearing and blurring. In this case, it is necessary to consider adjusting the number of superimposed frames. Then, the enhancement parameters of the current frame are smoothed in combination with the average parameters of the historical frames to reduce the "flicker" problem caused by the sudden change of the image signal between frames.

[0076] The algorithm involved in the patent method is written in C language, and the compilation and running environment is VisualStdio2017. The input image sequence is a 14-bit infrared original image sequence, and an 8-bit image sequence is obtained after being enhanced by the algorithm of the present invention. The algorithm of the present invention has strong stability and robustness in processing videos, and the processing method proposed by the algorithm has certain extensibility.

[0077] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for enhancing infrared image details based on video stream, characterized in that: The following steps are included S1, input N frames of infrared original images in the video stream, N ≥ 4, superimpose the Nth frame, the N-1th frame, the N-2th frame, the N-3th frame, ... images to obtain the original image data matrix; S2, bilateral filtering is performed on the original image data matrix, and the maximum and minimum values ​​are extracted and then smoothed in combination with the historical frame average parameters to obtain a parameter enhanced image; S3, taking the original image data matrix and the original image data matrix after bilateral filtering as input to perform detail enhancement, extracting the maximum value and combining it with the historical frame average parameter to perform smoothing to obtain a parameter enhanced image; S4, finally, linearly superimposing the parameter enhanced images of step S2 and step S3 to obtain an enhanced infrared original image.

2. The method for enhancing infrared image details based on video stream according to claim 1, characterized in that: When N=4, the step S1 is to first define the input image data matrix as A N-3 , A N-2 , A N-1 , A N , through formula D N =A N-3 +A N-2 +A N-1 +A N Superimpose to obtain the original image data matrix D N .

3. The method for enhancing infrared image details based on video stream according to claim 2, characterized in that: The step S2 is to convert the original image data matrix D N As input, bilateral filtering is performed and the image information of different dynamic ranges is separated using the following formula to obtain the base image that retains the image with a large dynamic range. The bilateral filter s(ix,jy)=g(x,y)·r(ix,jy) in the formula is a Gaussian low-pass filter kernel acting in the spatial domain. Grayscale difference filter kernel Where f(x,y) represents the grayscale value at the (x,y) coordinate.

4. The method for enhancing infrared image details based on video stream according to claim 3, characterized in that: In the steps S2 and S3, the unsharp mask image enhancement method is used to separate the small dynamic range detail image to obtain the detail image D. detail (i,j)=D N (i,j)-D base (i, j), then extract the parameters of each image and add the historical frame average parameter for weighting, and get the grayscale interval of the base image and detail image as and 5. The method for enhancing infrared image details based on video stream according to claim 4, characterized in that: Let the minimum and maximum values ​​of the base layer extracted from the image with frame number N be Then the grayscale interval used for grayscale mapping in this frame is The P value in the formula represents the number of base layer cache frames.

6. The method for enhancing infrared image details based on video stream according to claim 4, characterized in that: Let the small dynamic range detail map extracted by the unsharp mask be the maximum value Then the grayscale interval used for grayscale mapping in this frame is The P value in the formula represents the number of cached frames of the maximum parameter of the detail layer.

7. A method for enhancing infrared image details based on video stream according to claim 4, 5 or 6, characterized in that: The step S4 quantizes the 16-bit image obtained by superimposing multiple frames into an 8-bit visible image by enhancing the image details after parameter extraction and smoothing: (1) Let the 16-bit base image and detail image be and The corresponding statistical grayscale interval is First normalize the grayscale value according to the interval: Get (2) Grayscale mapping of the base image and detail image respectively: Where b and c are adjustment parameters, using formula f detail (x) = x γ Perform gamma transformation to achieve mapping of low grayscale intervals of detail images; (3) The calculated images are The image is quantized to 8-bit width using the following formula: In the two formulas, [] is the integer symbol, and the 8-bit base image is obtained after quantization. And details The base image and the detail image are linearly superimposed to form the output image of this algorithm:

Images