An image fusion method

Abstract

The present invention discloses an image fusion method, which seeks the two most similar pixel groups in two groups of images based on two fused images; performs lifting Haar transform on the two pixel groups to obtain two coefficient matrices; selects the coefficient with the largest absolute value in the two coefficient matrices as the fused high-frequency transform coefficient, and the mean of the two coefficient matrices as the fused low-frequency transform coefficient; combines the fused high-frequency transform coefficient and the fused low-frequency transform coefficient to obtain the fused transform coefficient; generates a fused image based on the fused transform coefficient. The fused image disclosed in the present invention can extract effective texture information from two complementary images, has a higher resolution than the fusion results of other methods, and has very ideal subjective visual quality. At the same time, the image fusion result is generally better than the existing image fusion methods in objective evaluation.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to an image fusion method. Background Art

[0002] Image fusion is the process of extracting the best information from each sensor and synthesizing it into a high-quality image when multiple sensors collect a large amount of redundant information about the same object. The fused image contains more comprehensive and richer information, is more in line with human or machine visual characteristics, and is more conducive to further analysis and processing of the image and automatic recognition of the target.

[0003] Infrared images can overcome dim light and complex weather obstacles to detect targets, but their images have low contrast and resolution. Visible light images can provide detailed and texture information of targets, but they are difficult to display well under complex backgrounds. The fusion of infrared and visible light can effectively integrate and explore their feature information, enhance scene understanding, highlight targets, and facilitate faster and more accurate detection of targets under hidden, camouflaged and confusing conditions. Its application range has spread to military, security monitoring and other fields.

[0004] The existing technology adopts an optimization method based on potential low-rank representation. Due to the characteristics of low-rank decomposition on a global scale, the texture information extracted by this method is always incomplete, which causes the subsequent fused image to lose some texture detail information and produce artifacts. The fusion strategy of taking the average of the significant parts after image decomposition makes the overall brightness of the image in the fusion result darker and the resolution lower. Summary of the invention

[0005] The purpose of this application is to solve at least one problem existing in the prior art, and to develop an image fusion method with a higher resolution than other methods, a very ideal subjective visual quality, and an image fusion result that is generally better than the existing image fusion methods in objective evaluation;

[0006] The above technical problems of the present disclosure are mainly solved by the following technical solutions:

[0007] In order to achieve the purpose of this application, the technical solutions adopted by this disclosure are as follows:

[0008] An image fusion method is used to perform image fusion, comprising the following steps:

[0009] S1. Based on the first image to be fused and the second image to be fused, respectively, seek the most similar first pixel group and second pixel group in the two groups of images.

[0010] S2, performing lifting Haar transform on the first similar pixel group and the second similar pixel group to obtain a first coefficient matrix and a second coefficient matrix;

[0011] Selecting coefficients with larger absolute values ​​in the first coefficient matrix and the second coefficient matrix as fused high-frequency transform coefficients, and using the average of the first coefficient matrix and the second coefficient matrix as fused low-frequency transform coefficients; wherein the low-frequency transform coefficient is the first coefficient of the coefficient matrix, and the high-frequency transform coefficient is the remaining coefficients of the coefficient matrix;

[0012] The fused high-frequency transform coefficient and the fused low-frequency transform coefficient are combined to obtain the fused transform coefficient;

[0013] S3. Generate a fused image based on the fused transformation coefficients.

[0014] Optionally, the specific execution method of step S1 is:

[0015] The first image to be fused and the second image to be fused are divided into a plurality of image blocks of the same size.

[0016] Extract the image blocks from the first image to be fused and the second image to be fused as the first reference block and the second reference block according to a specified step length, select the block with the largest standard deviation between the first reference block and the second reference block as the standard reference block,

[0017] Performing block matching in a neighborhood of a specified size centered on a standard reference block in the first image to be fused and the second image to be fused to obtain two groups of similar image blocks as a first similar image block group and a second similar block group;

[0018] Scan the first similar image block group and the second similar block group respectively and splice them into a first matrix and a second matrix;

[0019] Extracting a row of pixels from the first matrix and the second matrix in sequence as a first reference row and a second reference row, selecting a row with a large standard deviation from the first reference row and the second reference row as a standard reference row,

[0020] The Euclidean distances between the standard reference row and all other rows on the first matrix and the second matrix are calculated respectively, and a row on the first matrix and the second matrix having the smallest distance to the standard reference row is merged together with the reference row into the first pixel group and the second pixel group.

[0021] Optionally, the specific method of performing the lifting Haar transform on the first similar pixel group and the second similar pixel group is:

[0022] The lifting Haar transform includes a horizontal Haar transform and a vertical Haar transform which are performed separately;

[0023] The horizontal Haar transform is to perform weighted summation and weighted difference on the first similar pixel group and the second similar pixel group to obtain two rows of transformed coefficients respectively;

[0024] The longitudinal Haar transform is a weighted sum or difference between the first similar pixel group and the second similar pixel group.

[0025] Optionally, the specific execution method of step S3 is:

[0026] Performing an inverse Haar transform on the fused transformation coefficients to obtain a fused third similar pixel group;

[0027] All pixels in the third similar pixel group are placed back to their original positions in the matching process to generate a fused image.

[0028] Optionally, the first image to be fused or the second image to be fused is one of an infrared image, a visible light image, and a multimodal image.

[0029] The present invention solves the image fusion problem by selecting high-quality similar pixel groups in two images to be fused from two groups of similar pixel groups. Image fusion is an image fusion method of non-local model at the pixel level. In addition to being able to fuse infrared and visible light images well, it also has good effects on the fusion of multi-modal medical images and multi-focus image fusion. The fused image can extract effective texture information from two complementary images, and the resolution is higher than the fusion results of other methods. The subjective visual quality is very ideal. At the same time, the image fusion results are generally better than existing image fusion methods in objective evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 is a flow chart of the image fusion method in the present disclosure;

[0032] Figure 2 is a comparison of the image obtained by fusing infrared image and visible light image by the prior art method and the method disclosed in the present invention; wherein, Figure 2 a is the infrared image, Figure 2 b is the visible light image, Figure 2c is the result image of the fusion of infrared image and visible light image by the existing technology method, Figure 2 d is the result image of the fusion of infrared image and visible light image by the method disclosed in the present invention.

[0033] Figure 3 is a comparison of the resultant image of the fusion of two multimodal images by the prior art method and the method disclosed in the present invention; wherein, Figure 3 a is the first multimodal image, Figure 3 b is the second multimodal image, Figure 3 c is the result image of fusing multiple multimodal images using the existing technology method, Figure 3 d is the result image of fusing multiple multimodal images using the disclosed method.

[0034] Figure 4 is an example diagram of the result of the infrared and visible light image fusion method disclosed in the present invention; wherein, Figure 4 a is an infrared image, and Figure 4b is a visible light image. Figure 4 c is the result image fused by the disclosed method.

[0035] Figure 5 An example diagram of the results of the disclosed method for multimodal medical image fusion; wherein, Figure 5 a is the first multimodal image, Figure 5 b is the second multimodal image, Figure 5 c is the result image fused by the disclosed method. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present disclosure can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present disclosure.

[0037] Embodiment 1

[0038] See also Figure 1 , Figure 4 and Figure 5 As shown, an image fusion method is used to perform image fusion, comprising the following steps:

[0039] S1, find two groups of similar pixels;

[0040] Based on the first image to be fused and the second image to be fused, the most similar first pixel group and second pixel group in the two groups of images are respectively sought. The clearer blocks in the two images to be fused are selected in sequence according to the step size as reference blocks, and the positions of the reference blocks are recorded at the same time. A neighborhood of a specified size is divided in the two images to be fused with the position of the reference block as the center, and block matching is performed in the neighborhood to find the image block group that is most similar to the reference block as the first pixel group and the second pixel group.

[0041] Seeking similar pixel groups on the image to be fused ensures that the subsequent fusion is an operation at the non-local pixel level. The similar pixel group refers to the pixel group that is most similar to the pixel row in the image block within a certain neighborhood range, which simply and effectively extracts the significant information of the image. Wherein, the first image to be fused A or the second image to be fused B is one of an infrared image, a visible light image, and a multimodal image. The multimodal image can be a nuclear magnetic resonance image, a computer tomography image, a positron emission tomography image, a single photon emission computer tomography image, etc.

[0042] The specific implementation method can be:

[0043] The first image to be fused and the second image to be fused are divided into N image blocks of N1×N1,

[0044] At a specified step size N step Extracting image blocks of size N1×N1 from the first image to be fused and the second image to be fused as the first reference block and the second reference block respectively, selecting a block with a large standard deviation between the first reference block and the second reference block as a standard reference block,

[0045] Then, a block with a size of N centered on the standard reference block is formed in the first image to be fused and the second image to be fused. s ×N s Block matching is performed in the neighborhood to obtain two groups of N2 similar image blocks, which are used as the first similar image block group and the second similar block group;

[0046] Scan the first similar image block group and the second similar block group respectively and splice them into a (N1×N1)×N2 first matrix M1 and a second matrix M2;

[0047] A row of pixels is sequentially extracted from the first matrix M1 and the second matrix M2 as a first reference row and a second reference row, and a row with a large standard deviation is selected from the first reference row and the second reference row as a standard reference row.

[0048] The Euclidean distances between the standard reference row and all other rows on the first matrix M1 and the second matrix M2 are calculated respectively, and a row on the first matrix M1 and the second matrix M2 with the smallest distance to the standard reference row is merged with the reference row into the first pixel group and the second pixel group.

[0049] S2. Transform coefficient fusion

[0050] S21, performing a lifting Haar transform on the first similar pixel group and the second similar pixel group to obtain a first coefficient matrix and a second coefficient matrix; the lifting Haar transform is a separable Haar transform, which means dividing the two-dimensional Haar transform into two one-dimensional Haar transforms in the horizontal direction and in the vertical direction. The matrix of the similar pixel group found is not always a square matrix or a matrix that is exactly a power of 2, and the Haar transform coefficients cannot be used directly, so a separable Haar transform is performed on the similar pixel group matrix.

[0051] The specific method of performing the lifting Haar transform on the first similar pixel group and the second similar pixel group is:

[0052] The lifting Haar transform includes a horizontal Haar transform and a vertical Haar transform which are performed separately;

[0053] The horizontal Haar transform is to perform weighted summation and weighted difference on the first similar pixel group and the second similar pixel group to obtain two rows of transformed coefficients respectively;

[0054] The longitudinal Haar transform is a weighted sum or difference between the first similar pixel group and the second similar pixel group.

[0055] S22. According to wavelet theory, each coefficient matrix (the first transform coefficient matrix and the second transform coefficient matrix) can be divided into high-frequency coefficients (the remaining coefficients except the first coefficient of the coefficient matrix) and low-frequency coefficients (the first coefficient of the coefficient matrix), the high-frequency coefficients contain a large amount of texture information of the image, and the low-frequency coefficients contain smooth information; select the coefficients with large absolute values ​​in the first coefficient matrix and the second coefficient matrix as the fused high-frequency transform coefficients, and the mean of the first coefficient matrix and the second coefficient matrix is ​​used as the fused low-frequency transform coefficient;

[0056] S23, combining the fused high-frequency transform coefficient and the fused low-frequency transform coefficient to obtain a fused transform coefficient;

[0057] S3. Generate fused image

[0058] A fused image is generated based on the fused transform coefficients.

[0059] The specific execution method of step S3 may be:

[0060] Performing an inverse Haar transform on the fused transformation coefficients to obtain a fused third similar pixel group;

[0061] All pixels in the third similar pixel group are placed back according to the positions in the original matching process to generate a fused image. The positions in the original matching process are derived from recording the position of each block in the similar block group in the image when the image blocks are matched to form similar block groups. The position of each similar pixel group obtained after matching the similar block groups corresponds to the position of one of the blocks recorded above.

[0062] The image fusion method disclosed in the present invention can be applied to the fusion of infrared, CT, visible light and other medical images, and can also be applied to the fusion of military infrared equipment and visible light to increase the clarity of infrared images; or the image fusion of military infrared equipment and thermal imaging, etc.; and even video fusion, when video fusion requires the block matching blocks to be promoted from a two-dimensional matrix to a three-dimensional matrix.

[0063] In addition, when multiple images are to be fused, this method can be applied to fuse the images pairwise, ultimately achieving multi-image fusion.

[0064] Embodiment 2

[0065] See also Figure 2 As shown, the present disclosure uses MATLAB software to perform image fusion experiments on 12 groups of infrared and visible light images and 7 groups of multimodal medical images. For the infrared and visible light image fusion experiment, the parameter values ​​of the execution algorithm are exemplarily adopted: N1=8, N2=16, N S =13,N step =2.

[0066] The following table shows the infrared and visible light image fusion results Q abf Value comparison result;

[0067]

[0068]

[0069] It can be seen from the data in the table that the Q of the method results disclosed in the present invention is abf The values ​​are much higher than the results of the state-of-the-art methods for all images.

[0070] Embodiment 3

[0071] See also Figure 3 As shown, in this embodiment, for the multimodal medical image fusion experiment, the parameter values ​​of the execution algorithm are exemplarily adopted as follows: N1=16, N2=8, N S =33,N step =8.

[0072] The following table shows the multimodal image fusion results Q abf Value comparison result;

[0073]

[0074] It can be seen from the data in the table that the Q of the method results disclosed in the present invention is abf The values ​​are much higher than the results of the state-of-the-art methods for all images.

[0075] After comparative analysis, the technical solution of the present disclosure can achieve very ideal fusion results even for infrared images with very low resolution and visible light images with complex weak light backgrounds. In the fusion process, the texture detail information can be well preserved without generating block effects and artifacts. The present disclosure is not only effective for the fusion of infrared and visible light images, but also for the fusion of multimodal medical images and multi-focus image fusion. The present disclosure can help the existing image fusion algorithms to be applied in practice, so that they can obtain their practical value without just staying in theory.

[0076] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that are not conceived through creative work should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope defined in the claims.

Claims

1. An image fusion method for performing image fusion, characterized in that: The following steps are involved: S1, based on the first image to be fused and the second image to be fused, respectively seek the most similar first pixel group and second pixel group in the two groups of images; S2, performing lifting Haar transform on the first similar pixel group and the second similar pixel group to obtain a first coefficient matrix and a second coefficient matrix; Selecting coefficients with larger absolute values ​​in the first coefficient matrix and the second coefficient matrix as fused high-frequency transform coefficients, and using the average of the first coefficient matrix and the second coefficient matrix as fused low-frequency transform coefficients; wherein the low-frequency transform coefficient is the first coefficient of the coefficient matrix, and the high-frequency transform coefficient is the remaining coefficients of the coefficient matrix; The fused high-frequency transform coefficient and the fused low-frequency transform coefficient are combined to obtain the fused transform coefficient; S3, generating a fused image based on the fused transformation coefficients; The specific method of performing the lifting Haar transform on the first similar pixel group and the second similar pixel group is: The lifting Haar transform includes a horizontal Haar transform and a vertical Haar transform which are performed separately; The horizontal Haar transform is to perform weighted summation and weighted difference on the first similar pixel group and the second similar pixel group to obtain two rows of transformed coefficients respectively; The longitudinal Haar transform is a weighted sum or difference between the first similar pixel group and the second similar pixel group.

2. The image fusion method according to claim 1, characterized in that: The specific execution method of step S1 is: The first image to be fused and the second image to be fused are divided into a plurality of image blocks of the same size. Extract the image blocks from the first image to be fused and the second image to be fused as the first reference block and the second reference block according to a specified step length, select the block with the largest standard deviation between the first reference block and the second reference block as the standard reference block, Performing block matching in a neighborhood of a specified size centered on a standard reference block in the first image to be fused and the second image to be fused to obtain two groups of similar image blocks as a first similar image block group and a second similar block group; Scan the first similar image block group and the second similar block group respectively and splice them into a first matrix and a second matrix; Extracting one row of pixels from the first matrix and the second matrix in sequence as a first reference row and a second reference row, and selecting a row with a larger standard deviation from the first reference row and the second reference row as a standard reference row; The Euclidean distances between the standard reference row and all other rows on the first matrix and the second matrix are calculated respectively, and a row on the first matrix and the second matrix having the smallest distance to the standard reference row is merged together with the reference row into the first pixel group and the second pixel group.

3. The image fusion method according to claim 2, characterized in that: The specific implementation method of step S3 is: Performing an inverse Haar transform on the fused transformation coefficients to obtain a fused third similar pixel group; All pixels in the third similar pixel group are placed back to their original positions in the matching process to generate a fused image.

4. The image fusion method according to any one of claims 1 to 3, characterized in that: The first image to be fused or the second image to be fused is one of an infrared image, a visible light image, and a multimodal image.

Images