Image fusion method based on information recycling and frequency domain gaussian weight pyramid

Abstract

The application discloses an image fusion method based on information recycling and frequency domain Gaussian weight pyramid, and belongs to the technical field of image fusion. The method first adopts a Gaussian weight sampling method to perform up-sampling and down-sampling, and in the up-sampling and down-sampling process, a frequency domain filter is used to replace a space domain filter to perform filtering, so as to construct a frequency domain Gaussian weight pyramid. Secondly, an image fusion pyramid is constructed by fusing infrared and visible light images with different resolutions by using a pulse ignition frequency map, and a recycling pyramid is constructed for the lost detail information in the fusion process. Finally, the result images are respectively reconstructed by using the two pyramids, and the final fusion image is obtained by superimposition. The fusion method fully considers the changes of high-frequency and low-frequency information of the image, recycles the discarded part of information in the image fusion process, reduces the problems of blurred image visual effect and information loss existing in other fusion methods, and improves the quality of the fusion image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image fusion, and particularly relates to an image fusion method based on information recycling and frequency domain Gaussian weight pyramid. BACKGROUND

[0002] Image fusion is one of important branches of information fusion, and can be specifically divided into multi-focus image fusion, multi-exposure image fusion, infrared and visible light image fusion, remote sensing image fusion, and medical image fusion, etc. In the development process of image fusion, infrared and visible light image fusion is one of the research focuses, which fuses two types of images obtained by heterogeneous sensors, so as to provide multi-level information description of the same scene from different angles. Specifically, the imaging process of infrared image is based on thermal radiation principle, and thus has the shortcomings of more image noise, poor resolution, low contrast, and blurred visual effect, etc. The infrared imaging process does not have any requirements on light conditions. In comparison, the acquisition of visible light image needs better light conditions. Generally, the visible light image has higher spatial resolution and quite a lot of details and light-dark contrast. The complementary characteristics of infrared image and visible light image can generate a more comprehensive and detailed information fusion image through a fusion algorithm, which is convenient for subsequent engineering application.

[0003] In recent years, the research on infrared and visible light image fusion has made great progress, and many excellent fusion algorithms have been generated. According to the information extraction level from low to high, the existing algorithms can be divided into three categories: pixel-level image fusion, feature-level image fusion, and decision-level image fusion.

[0004] Pixel-level fusion directly fuses the features of different images according to a given fusion rule, and finally generates a fusion image. It retains the most original information and has the highest fusion accuracy, but has the shortcomings of large amount of calculation information and high computational complexity.

[0005] Feature-level image fusion first pre-processes the images, then extracts the edge, texture and other feature information of the images through a given model, and finally generates a fusion image by selecting and fusing the feature information according to different fusion rules. This fusion method extracts the feature information of the original image, and injects the detail information of the image into the fusion image, so as to have good visual effect.

[0006] Decision-level fusion has independently completed the classification, identification and other decision tasks of each image before image fusion, and then comprehensively analyzes each independent decision result to generate a global optimal decision result and form a fusion image. This fusion method has the advantages of high flexibility and strong fault tolerance, and is more suitable for image fusion tasks under non-registration conditions. However, decision-level image fusion needs to first make decision and judgment on each image, which leads to more image pre-processing tasks and high algorithm complexity. SUMMARY

[0007] The present application provides an image fusion method based on information recycling and frequency domain Gaussian weight pyramid, aiming at solving the problems of blurred visual effect and information loss in the traditional infrared and visible light image fusion process.

[0008] The specific steps of the fusion method are as follows:

[0009] Step S1, using frequency domain filtering to process the original infrared image I inf (i,j) and the visible light image I vis (i,j), to obtain the same size blurred infrared image B inf (i,j) and the blurred visible light image B vis (i,j), wherein the image variable i=1, 2, …, N, j=1, 2, …, M, M and N are the row number and column number of the original image respectively;

[0010] Step S2, Gaussian weight down-sampling is performed on the blurred infrared image B inf (i,j) and the blurred visible light image B vis (i,j) to obtain the low-resolution image D inf (x,y) and D vis (x,y), x=1, 2, …, N / 2, y=1, 2, …, M / 2;

[0011] Step S3, the low-resolution image D inf (x,y) and D vis (x,y) obtained in step S2 are subjected to one Gaussian weight up-sampling and frequency domain filtering to obtain the rough image U inf (i,j) and U vis (i,j), respectively, U inf (i,j) and B inf (i,j) are subtracted, U vis (i,j) and B vis (i,j) are subtracted, so as to obtain the high-frequency information image R inf (i,j) and R vis (i,j), i=1, 2, …, N, j=1, 2, …, M;

[0012] Step S4, the high-frequency information images R inf (i,j) and R vis (i,j) are selected and fused by using the pulse ignition frequency image, and the above steps are iterated to construct the fusion pyramid FP;

[0013] Step S5, the high frequency information before and after fusion is subtracted to obtain the discarded detail information, and the above steps are iterated to construct the information recovery pyramid RP;

[0014] Step S6, two images are reconstructed, which are the fusion pyramid reconstructed image C FP (i, j) and the recovery pyramid reconstructed image C RP (i, j), i = 1, 2, …, N, j = 1, 2, …, M;

[0015] Step S7, the two reconstructed images C FP (i, j) and C RP (i, j) are superimposed to obtain the final fusion image Y(i, j), i = 1, 2, …, N, j = 1, 2, …, M.

[0016] Preferably, in the step S1, the specific implementation process of the frequency domain filtering processing is: using two-dimensional fast Fourier transform to convert the source image from the spatial domain to the frequency domain, which is physically said to be transforming the gray scale distribution function of the image into the frequency distribution function. Then move the frequency distribution function to the frequency spectrum center, and construct a frequency domain low-pass filter with the same size as the image to process the image, and the transfer function of the frequency domain low-pass filter is: Where F0 is a constant, the specific value changes with the size of the image, n = 3, F(u, v) is a normalized coordinate matrix, and the calculation formula is X(u, v) is the coordinate matrix of the image, u = 1, 2, …, N, v = 1, 2, …, M, and M and N are the number of rows and columns of the image subjected to frequency domain filtering processing.

[0017] Preferably, in the step S2, the calculation formula of the Gaussian weight down-sampling is:

[0018] Where "*" is the convolution operator, and δ(x, y) is defined as:

[0019] Preferably, in the step S3, the Gaussian weight up-sampling formula is: U(i, j) = D(x, y) * δ(i / 2-x, j / 2-y), i = 1, 2, …, N, j = 1, 2, …, M, x = 1, 2, …, N / 2, y = 1, 2, …, M / 2.

[0020] Preferably, in the step S4, the formula of the pulse ignition is The generation frequency formula is Where "*" is the convolution operator, i = 1, 2, …, N, j = 1, 2, …, M, and A is a full 1 matrix with size t x t.

[0021] The present application has the following advantages compared with the prior art:

[0022] Firstly, the present application uses a frequency domain filter to construct a pyramid. Since in many cases, the frequency domain representation is more sparse or the energy is more concentrated, the filtering effect is more significant compared with the spatial domain filter. In the spatial domain, the image is a gray distribution function. In the filtering process, part of the low-frequency information is filtered together with the high-frequency information, resulting in a serious loss of information. Frequency domain filtering is performed on the frequency distribution function with a clear distinction between high-frequency information and low-frequency information, thus eliminating this phenomenon. The rationality of the filter is increased, and it is easier to filter out the gray change information that is higher or lower than a certain frequency, while keeping other gray change information unchanged.

[0023] Secondly, the frequency domain Gaussian weight pyramid used in the present application is different from the existing Laplacian pyramid method. The existing Laplacian pyramid uses a relatively simple every-other-point sampling method for downsampling, that is, one point is taken every k points in each row and column (generally k = 2). The advantage is that the calculation is simple and easy to implement, but it produces obvious jagged edges and poor visual effects. The frequency domain Gaussian weight pyramid proposed in the present application is different from the traditional pyramid. Its core is to use Gaussian weight interpolation principle for sampling and frequency domain filter for filtering. The combination of the two can make the image texture smooth while being upsampled and downsampled, thus eliminating the jagged edges in the pyramid image and better reflecting the characteristics of the pyramid. While preserving the complete high and low frequency information of the image, different scale and more complete scaled images are obtained.

[0024] Thirdly, the present application maximizes the preservation of source image information through information recycling. In the image fusion process, the discarded information is recycled through certain methods, greatly reducing the information loss caused by the algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The present application proposes an image fusion method based on information recycling and frequency domain Gaussian weight pyramid, and its implementation step flow chart is as follows. DETAILED DESCRIPTION

[0026] In order to facilitate understanding and implementation of the present application, the technical solutions of the present application are further described in detail in combination with the drawings and examples in the specification. The described examples are only a part of the examples of the present application, not all examples. Based on the examples in the present application, all other examples obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0027] The present application proposes an image fusion method based on information recycling and frequency domain Gaussian weight pyramid, and its implementation step flow chart is as follows.Figure 1 as shown.

[0028] In combination Figure 1 , the image fusion method based on information recycling and frequency domain Gaussian weight pyramid of the present application comprises the following steps:

[0029] Step S1, processing the original infrared and visible light images to be fused. Before processing the original images to be fused, the resolution thereof needs to be adjusted to ensure that the number of rows and columns of the images after sampling is still an integer. The original infrared image is marked as and the original visible light image is marked as In the above notations, the subscript "inf" represents the infrared image, the subscript "vis" represents the visible light image, and the superscript "0" represents the corresponding image at the bottom layer of the pyramid model, and similarly, the image with halved resolution after one sampling is located at the first layer of the pyramid, which is represented by the superscript "1", and so on. First, the and are respectively subjected to two-dimensional fast Fourier transform to convert from the spatial domain to the frequency domain, and then subjected to Gaussian low-pass filtering in the frequency domain, and then subjected to two-dimensional inverse fast Fourier transform to return to the spatial domain to obtain the blur images with the same size and with high-frequency information filtered out and wherein the image variables i=1, 2, …, N, j=1, 2, …, M, and M and N are respectively the number of rows and columns of the original images.

[0030] The specific implementation process of the frequency domain filtering processing is as follows: two-dimensional fast Fourier transform is adopted to convert the source image from the spatial domain to the frequency domain, which is, in a physical sense, to transform the gray scale distribution function of the image into the frequency distribution function. Then, the frequency distribution function is moved to the center of the frequency spectrum, and a frequency domain low-pass filter with the same size as the image is constructed to process the image, and the transfer function of the frequency domain low-pass filter is: wherein F0 is a constant, the specific value of which varies with the size of the image, n=3, and F(u, v) is a normalized coordinate matrix, and the calculation formula thereof is X(u, v) is the coordinate matrix of the image, u=1, 2, …, N, v=1, 2, …, M, and M and N are respectively the number of rows and columns of the image subjected to the frequency domain filtering processing.

[0031] Step S2, performing Gaussian weight downsampling on the blur images and to obtain low-resolution images with half the number of rows and columns of the original images and The calculation formula of the Gaussian weight downsampling is: wherein "*" is a convolution operator, and δ(x, y) is defined as: x = 1, 2, …, N / 2, y = 1, 2, …, M / 2.

[0032] Step S3, the and are respectively subjected to Gaussian weight upsampling and frequency domain filtering to obtain coarse images and and are respectively subtracted from and , so as to obtain high-frequency information images of sampling loss and i = 1, 2, …, N, j = 1, 2, …, M.

[0033] The Gaussian weight upsampling formula is: U(i, j) = D(x, y)·δ(i / 2-x, j / 2-y), i = 1, 2, …, N, j = 1, 2, …, M, x = 1, 2, …, N / 2, y = 1, 2, …, M / 2

[0034] Step S4, the high-frequency information images of sampling loss and are fused by the method of pulse ignition frequency image, which is to generate pulses according to the size relationship between the internal output and the threshold value, and to accumulate and count the ignition frequency of each pixel point according to the pulse generation condition of each pixel point.

[0035] The formula of pulse ignition is The formula of generating frequency is Wherein “*” is the convolution operator symbol, i = 1, 2, …, N, j = 1, 2, …, M, and A is a full 1 matrix with a size of t x t. In the embodiment, the value of t is 3.

[0036] The fusion process of pulse ignition frequency image in the above step S4 is as follows: first, the gray values of each point of and are traversed in turn, and when the value of is greater than or equal to the value of , it is judged as “ignition success”, otherwise “ignition failure”, so as to obtain the pulse ignition image G 0 (i, j). Then, the pulse ignition image G 0 (i, j) is iteratively convolved by using a 3*3 full 1 matrix template, and the ignition frequency image H 0 (i, j) of more than half of the points in the template that ignite successfully is calculated. Finally, the two high-frequency information images are fused by using the formula to obtain the 0th layer of the fusion pyramid FP

[0037] ​Step S5, the high frequency information before and after fusion is subtracted to obtain the discarded detail information, and the above steps are iterated to construct the information recovery pyramid RP. Specifically, the formula is used to obtain the discarded detail information in the original fusion process as the 0th layer of the information recovery pyramid RP

[0038] The above steps are iterated repeatedly to calculate and When the image resolution is reduced to a certain extent (generally, iteration is performed 4 times, i.e. n=4), the iteration is ended, and the top layer of the pyramid is constructed. The top layer of the pyramid should be the image with the lowest resolution, and the formula is used for direct mean fusion, which is also the top layer of the fusion pyramid and the information recovery pyramid.

[0039] Step S6, the fusion image and the information recovery image are reconstructed through the constructed fusion pyramid FP and the information recovery pyramid RP. Specifically, the top layer image B n+1 is directly up-sampled with the Gaussian weight, and then subjected to frequency domain Gaussian low-pass filtering, and then the detail information P n of the nth layer is added. Iteration is repeated to the 0th layer, i.e. the reconstruction of the image is completed, i.e. two images are reconstructed, which are the fusion pyramid reconstruction image C FP and the recovery pyramid reconstruction image C RP , i=1, 2, …, N, j=1, 2, …, M.

[0040] Step S7, considering that the information recovery image and the fusion image have completed image fusion at different levels, the two reconstructed images C FP and C RP are superimposed, i.e. the mean value of the corresponding pixel points is obtained to obtain the final fusion image Y(i, j).

[0041] The specific implementation process of the information recovery in image fusion is that the discarded image information in each image fusion process is retained, and the discarded information is reused in the subsequent fusion process. Since the image may contain noise, blur and other factors affecting the visual effect of the image, the image fusion algorithm is used to combine the advantages of multiple source images to obtain a better fusion image, but in this process, part of the information is inevitably lost, and the noise and part of the useful high frequency information of the image are discarded. In the present application, after the noise in the image fusion process is discarded, the discarded high frequency information is recovered, a new information recovery pyramid is constructed, and the same operation as the fusion image is performed to reconstruct the information recovery image, and then the final fusion image is obtained by directly adding the preliminary fusion image in the last stage.

[0042] It should be noted that the above embodiments can be freely combined as needed. The above only describes the preferred embodiments of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. An image fusion method based on information recycling and frequency domain Gaussian weight pyramid, characterized in that: The specific steps are as follows: Step S1, using frequency domain filtering to process the original infrared image and the visible light image to obtain an infrared image blurred by the same size and a blurred visible light image , where the image variables , , M and N are the row number and column number of the original image, respectively; Step S2, Gaussian weight down-sampling is performed on the blurred infrared image and the blurred visible light image to obtain a low-resolution image after down-sampling and , , ; Step S3, performing one time Gaussian weight up-sampling and frequency domain filtering on the low-resolution image obtained in step S2 to obtain a rough image and performing one time Gaussian weight up-sampling and frequency domain filtering on the low-resolution image obtained in step S2 to obtain a rough image and respectively subtracting from , from , so as to obtain an image of high-frequency information lost after image sampling and ; Step S4: Analyze the high-frequency information of the two images. and Selective fusion is performed using pulse ignition frequency images. The pulse ignition frequency image fusion process is as follows: First, iterate through the images sequentially. and The grayscale value of each point, when The value is greater than or equal to When the value is specified, it is determined as "ignition successful"; otherwise, it is "ignition failed," thus obtaining the pulse ignition diagram. Then, adopt The all-1 matrix template will display the pulse ignition pattern. Iterative convolution is used to calculate the ignition frequency map where more than half of the ignitions within the template are successful. Finally, use the formula. By fusing two high-frequency information images, a fusion pyramid is obtained. Layer 0 Iterate through steps S1-S3, accumulating 4 iterations, to construct the fusion pyramid. ; Step S5, the high frequency information before and after fusion is subtracted to obtain the discarded detail information, the above steps S1-S4 are iterated, and the iteration is accumulated for 4 times to construct the information recycling pyramid ; Step S6, reconstructing two images, respectively, the fusion pyramid reconstructed image and the recovery pyramid reconstructed image and the recovery pyramid reconstructed image , , ; Step S7, superimposing the two reconstructed images and stacking, i.e. averaging the corresponding pixels to obtain the final fused image .

2. The image fusion method based on information recycling and frequency domain Gaussian weight pyramid of claim 1, wherein: The specific implementation process of the frequency domain filtering processing in the step S1 is: adopting two-dimensional fast Fourier transform, converting the source image from the spatial domain to the frequency domain, moving the frequency distribution function to the frequency spectrum center, constructing a frequency domain low-pass filter with the same size as the image to process the image, and the transfer function of the frequency domain low-pass filter is: wherein is a constant, , is a normalized coordinate matrix, and the calculation formula thereof is , is a coordinate matrix of the image, , , M and N are the number of rows and columns of the image, respectively, on which the frequency domain filtering process is performed.

3. The image fusion method based on information recycling and frequency domain Gaussian weight pyramid of claim 1, wherein: The formula for the Gaussian weight down-sampling in the step S2 is: wherein is the convolution operator, is defined as: .

4. The image fusion method based on information recycling and frequency domain Gaussian weight pyramid of claim 1, wherein: The formula for the Gaussian weight up-sampling in the step S3 is: The formula for the Gaussian weight up-sampling in the step S3 is: wherein , , , .

5. The image fusion method based on information recycling and frequency domain Gaussian weight pyramid of claim 1, wherein: The formula of the pulse ignition in step S4 is The formula of the generating frequency is , wherein is a convolution operator, , A is a full one matrix with the size of , and the value of the parameter is 3.

Images