Image fusion method and device, equipment and medium

Abstract

The embodiment of the invention discloses an image fusion method and device, equipment and a medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring a first image and a second image; respectively performing wavelet decomposition on the first image and the second image to obtain a low-frequency coefficient and a high-frequency coefficient of the first image and a low-frequency coefficient and a high-frequency coefficient of the second image; according to the low-frequency coefficient and the high-frequency coefficient of the first image and the low-frequency coefficient and the high-frequency coefficient of the second image, generating a fused low-frequency coefficient and a fused high-frequency coefficient through non-negative matrix factorization; and performing wavelet inverse transformation on the fused low-frequency coefficient and the fused high-frequency coefficient to obtain a fused third image. Therefore, according to the image fusion method disclosed by the embodiment of the invention, the clear areas of the first image and the second image can be integrated by fusing the first image and the second image, so that the third image with more comprehensive focus and more complete information is generated, and the utilization rate of the information in the image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image fusion method, device, equipment and medium. Background Art

[0002] The core goal of the single-aberration adjustable focus lens technology that imitates the human eye lens is to imitate the precise focusing ability of the human eye lens to achieve a more flexible and efficient zoom effect in photography. This lens dynamically adjusts its physical properties, such as refractive index or geometric shape, to provide the camera with the ability to shoot at multiple focal lengths during continuous zooming, ensuring that each photo can capture unique scenes at different focal lengths.

[0003] However, due to the limited zoom range and focusing accuracy of the lens, under extreme shooting conditions, especially when the subject is too close or too far from the camera, the captured image may have insufficient clarity in some areas or even complete loss of details in some areas, making it difficult to meet high-precision shooting requirements. Summary of the invention

[0004] Based on the above problems, the present application provides an image fusion method, device, equipment and medium that can meet the needs of high-precision shooting.

[0005] The embodiments of the present application disclose the following technical solutions:

[0006] In a first aspect, the present application discloses an image fusion method, the method comprising:

[0007] Acquire a first image and a second image, wherein the first image and the second image have different focal positions;

[0008] Performing wavelet decomposition on the first image and the second image respectively to obtain low-frequency coefficients and high-frequency coefficients of the first image, and low-frequency coefficients and high-frequency coefficients of the second image;

[0009] According to the low-frequency coefficients and high-frequency coefficients of the first image and the low-frequency coefficients and high-frequency coefficients of the second image, generating fused low-frequency coefficients and high-frequency coefficients through non-negative matrix decomposition;

[0010] Performing inverse wavelet transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a fused third image.

[0011] Optionally, the generating fused low-frequency coefficients and high-frequency coefficients by non-negative matrix decomposition according to the low-frequency coefficients and high-frequency coefficients of the first image and the low-frequency coefficients and high-frequency coefficients of the second image includes:

[0012] Generate a first matrix according to the low-frequency coefficients and high-frequency coefficients of the first image and the low-frequency coefficients and high-frequency coefficients of the second image;

[0013] Performing non-negative matrix decomposition on the first matrix to obtain a basis matrix and a coefficient matrix;

[0014] The fused low-frequency coefficients and fused high-frequency coefficients are obtained by reconstructing the fused coefficients of the base matrix and the coefficient matrix.

[0015] Optionally, the generating fused low-frequency coefficients and fused high-frequency coefficients by reconstructing the fused coefficients of the base matrix and the coefficient matrix includes:

[0016] According to the low-frequency coefficients in the coefficient matrix, a first coefficient value is obtained by a weighted average fusion method;

[0017] According to the high frequency coefficients in the coefficient matrix, a second coefficient value is obtained by a maximum fusion method;

[0018] The first coefficient value and the second coefficient value are multiplied by the base matrix respectively to obtain a fused low-frequency coefficient and a fused high-frequency coefficient.

[0019] Optionally, the method further includes a fusion iteration step, and the fusion iteration step includes:

[0020] When there is a fourth image having a different focal position from the fused third image, performing wavelet decomposition on the third image and the fourth image to obtain low-frequency coefficients and high-frequency coefficients of the third image, and low-frequency coefficients and high-frequency coefficients of the fourth image;

[0021] Generate a secondary fused low-frequency coefficient and a secondary fused high-frequency coefficient according to the low-frequency coefficient and the high-frequency coefficient of the third image and the low-frequency coefficient and the high-frequency coefficient of the fourth image;

[0022] Performing inverse wavelet transform on the secondary fused low-frequency coefficients and the secondary fused high-frequency coefficients to obtain a fused fifth image;

[0023] The fusion iteration step is repeated until there is no image to be fused whose focal position is different from that of the fused image.

[0024] Optionally, performing wavelet decomposition on the first image and the second image respectively to obtain low-frequency coefficients and high-frequency coefficients of the first image, and low-frequency coefficients and high-frequency coefficients of the second image, includes:

[0025] Performing a two-dimensional discrete wavelet transform on the first image and the second image respectively to obtain a low frequency subband, a horizontal high frequency subband, a vertical high frequency subband, and a diagonal high frequency subband of the first image, and a low frequency subband, a horizontal high frequency subband, a vertical high frequency subband, and a diagonal high frequency subband of the second image, wherein the two-dimensional discrete wavelet transform includes filtering in the horizontal direction and the vertical direction;

[0026] Extracting low-frequency coefficients of the first image and low-frequency coefficients of the second image from the low-frequency subband of the first image and the low-frequency subband of the second image respectively;

[0027] The high-frequency coefficients of the first image and the high-frequency coefficients of the second image are respectively extracted from the horizontal high-frequency subband, vertical high-frequency subband and diagonal high-frequency subband of the first image and the horizontal high-frequency subband, vertical high-frequency subband and diagonal high-frequency subband of the second image.

[0028] Optionally, acquiring the first image and the second image includes:

[0029] Acquire a first original image and a second original image, wherein the first original image and the second original image have different focal positions;

[0030] The first original image and the second original image are registered by at least one of feature point matching, affine transformation and projective transformation to obtain the first image and the second image, and the first image and the second image are aligned in spatial position.

[0031] Optionally, after performing the inverse wavelet transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a fused third image, the method further includes:

[0032] At least one of edge processing, color correction and smoothing processing is performed on the fused third image to obtain a processed third image.

[0033] In a second aspect, the present application discloses an image fusion device, the device comprising: an image acquisition module, a wavelet decomposition module, a coefficient generation module and an image fusion module;

[0034] The image acquisition module is used to acquire a first image and a second image, wherein the focus positions of the first image and the second image are different;

[0035] The wavelet decomposition module is used to perform wavelet decomposition on the first image and the second image respectively to obtain low-frequency coefficients and high-frequency coefficients of the first image, and low-frequency coefficients and high-frequency coefficients of the second image;

[0036] The coefficient generation module is used to generate fused low-frequency coefficients and high-frequency coefficients according to the low-frequency coefficients and high-frequency coefficients of the first image and the low-frequency coefficients and high-frequency coefficients of the second image through non-negative matrix decomposition;

[0037] The image fusion module is used to perform inverse wavelet transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a fused third image.

[0038] Optionally, the coefficient generation module includes: a first generation submodule, a second generation submodule and a third generation submodule;

[0039] The first generating submodule is used to generate a first matrix according to the low-frequency coefficients and high-frequency coefficients of the first image and the low-frequency coefficients and high-frequency coefficients of the second image;

[0040] The second generating submodule is used for performing non-negative matrix decomposition on the first matrix to obtain a base matrix and a coefficient matrix;

[0041] The third generating submodule is used for reconstructing the coefficients by fusing the base matrix and the coefficient matrix to obtain fused low-frequency coefficients and fused high-frequency coefficients.

[0042] Optionally, the third generation submodule includes: an acquisition submodule and a fusion submodule;

[0043] The acquisition submodule is used to acquire the first coefficient value by a weighted average fusion method according to the low-frequency coefficients in the coefficient matrix;

[0044] The acquisition submodule is further used to acquire a second coefficient value by a maximum fusion method according to the high frequency coefficients in the coefficient matrix;

[0045] The fusion submodule is used to multiply the first coefficient value and the second coefficient value by the base matrix respectively to obtain a fused low-frequency coefficient and a fused high-frequency coefficient.

[0046] Optionally, the wavelet decomposition module includes: a first decomposition submodule and a second decomposition submodule;

[0047] The first decomposition submodule is used to perform two-dimensional discrete wavelet transform on the first image and the second image respectively to obtain a low frequency subband, a horizontal high frequency subband, a vertical high frequency subband and a diagonal high frequency subband of the first image, and a low frequency subband, a horizontal high frequency subband, a vertical high frequency subband and a diagonal high frequency subband of the second image, wherein the two-dimensional discrete wavelet transform includes filtering in the horizontal direction and the vertical direction;

[0048] The second decomposition submodule is used to extract low-frequency coefficients of the first image and low-frequency coefficients of the second image from the low-frequency subband of the first image and the low-frequency subband of the second image respectively;

[0049] The second decomposition sub-module is also used to extract the high-frequency coefficients of the first image and the high-frequency coefficients of the second image from the horizontal high-frequency sub-band, vertical high-frequency sub-band and diagonal high-frequency sub-band of the first image, and the horizontal high-frequency sub-band, vertical high-frequency sub-band and diagonal high-frequency sub-band of the second image, respectively.

[0050] Optionally, the image acquisition module includes: a first acquisition submodule and a second acquisition submodule;

[0051] The first acquisition submodule is used to acquire a first original image and a second original image, wherein the first original image and the second original image have different focal positions;

[0052] The second acquisition submodule is used to align the first original image and the second original image by at least one of feature point matching, affine transformation and projective transformation to obtain the first image and the second image, and the first image and the second image are aligned in spatial position.

[0053] Optionally, the device further comprises: a post-processing module;

[0054] The post-processing module is used to perform at least one of edge processing, color correction and smoothing processing on the fused third image to obtain a processed third image.

[0055] Compared with the prior art, this application has the following beneficial effects:

[0056] The embodiment of the present application discloses an image fusion method, device, equipment and medium, the method comprising: acquiring a first image and a second image; performing wavelet decomposition on the first image and the second image respectively to obtain low-frequency coefficients and high-frequency coefficients of the first image, and low-frequency coefficients and high-frequency coefficients of the second image; generating fused low-frequency coefficients and high-frequency coefficients through non-negative matrix decomposition according to the low-frequency coefficients and high-frequency coefficients of the first image, and the low-frequency coefficients and high-frequency coefficients of the second image; performing inverse wavelet transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a fused third image. Thus, the image fusion method disclosed in the embodiment of the present application can integrate their respective clear areas by fusing the first image and the second image, thereby generating a third image with more comprehensive focus and more complete information. This fusion not only improves the utilization rate of information in the image, but also significantly improves the visual effect and practical application value of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Figure 1 A flowchart of the first image fusion method provided in an embodiment of the present application;

[0059] Figure 2 A flowchart of a second image fusion method provided in an embodiment of the present application;

[0060] Figure 3 A schematic diagram of image fusion provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of image fusion based on a one-way differential focus-adjustable lens that simulates human eye lens provided in an embodiment of the present application;

[0062] Figure 5 A schematic diagram of an image fusion device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] As described above, due to the limited zoom range and focusing accuracy of the single-aberration adjustable focus lens that simulates the human eye lens, under extreme shooting conditions, especially when the subject is too close or too far from the camera, the captured image may have insufficient clarity in some areas, or even complete loss of details in some areas, making it difficult to meet high-precision shooting requirements.

[0064] After research, the inventors have proposed an image fusion method, device, equipment and medium. The image fusion method disclosed in the embodiment of the present application can integrate the respective clear areas of the first image and the second image by fusing them, thereby generating a third image with more comprehensive focus and more complete information. This fusion not only improves the utilization rate of information in the image, but also significantly improves the visual effect and practical application value of the generated image. Whether in medical imaging, remote sensing monitoring, photography art or other fields that require high-quality image fusion, the image fusion method disclosed in the embodiment of the present application has important application value.

[0065] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0066] See also Figure 1 , which is a flow chart of the first image fusion method provided in the embodiment of the present application. The method comprises:

[0067] S101: Acquire a first image and a second image, wherein the focus positions of the first image and the second image are different.

[0068] The difference between the first image and the second image is that their focus positions are different, which is caused by different focus positions used when shooting or generating the images.

[0069] Specifically, when an image is captured, a specific focus position is usually selected to ensure that a certain part of the image (such as the foreground or background) is presented clearly. If different focus positions are selected when two images are captured, the two images will highlight different parts. For example, in the first image, the foreground may be very clear because the focus position is set on the foreground; while in the second image, the background may be clearer because the focus position is set on the background.

[0070] S102: performing wavelet decomposition on the first image and the second image respectively to obtain low-frequency coefficients and high-frequency coefficients of the first image, and low-frequency coefficients and high-frequency coefficients of the second image.

[0071] When performing wavelet decomposition on an image, you need to first select a suitable wavelet basis function, such as Haar wavelet, Daubechies wavelet, Symlets wavelet, etc.

[0072] Secondly, a two-dimensional filter bank is applied to the entire image (i.e., the two-dimensional signal matrix). The two-dimensional filter bank usually includes a low-pass-low-pass filter (used to generate approximate components or low-frequency coefficients), a high-pass-low-pass filter (used to generate detail components or part of high-frequency coefficients in the horizontal direction), a low-pass-high-pass filter (used to generate detail components or another part of high-frequency coefficients in the vertical direction), and a high-pass-high-pass filter (used to generate detail components or another part of high-frequency coefficients in the diagonal direction). By performing a convolution operation on the image through the two-dimensional filter bank and downsampling the convolution result, four sub-bands can be obtained: a low-frequency sub-band (approximate component) and three high-frequency sub-bands (corresponding to detail components in the horizontal, vertical, and diagonal directions, respectively).

[0073] Then, based on the low-frequency sub-band and the high-frequency sub-band, the low-frequency coefficients and high-frequency coefficients corresponding to the image can be obtained. Among them, the low-frequency sub-band represents the main contour and low-frequency information of the image. The high-frequency sub-band contains high-frequency detail information such as the edge and texture of the image.

[0074] S103: Generate fused low-frequency coefficients and high-frequency coefficients by non-negative matrix decomposition according to the low-frequency coefficients and high-frequency coefficients of the first image and the low-frequency coefficients and high-frequency coefficients of the second image.

[0075] When fusing low-frequency coefficients, a weighted average fusion method can be used to generate the fused low-frequency coefficients. The weighted average fusion method generates the fused low-frequency coefficients by weighted averaging the low-frequency coefficients of the first image and the second image. The specific value of the weight can be determined according to different application scenarios and image characteristics. For example, if the structural information of two images in a certain area is very important, then the low-frequency coefficients of the two areas can be given equal weights; if the structural information of one image in a certain area is obviously better than that of the other image, then the low-frequency coefficients of the area can be given a higher weight. The purpose of the weighted average fusion method is to enable the fused low-frequency coefficients to retain the most important structural information of the two images while reducing possible blur or distortion. Through reasonable weight distribution, the contribution of the two images in the fusion result can be balanced, making the fused image more reasonable and clear in structure.

[0076] When fusing high-frequency coefficients, the maximum fusion method can be used to generate the fused high-frequency coefficients. The maximum fusion method compares the high-frequency coefficients of the first image and the second image at the same position, and selects the larger one as the fused high-frequency coefficient. Since the high-frequency coefficients contain the detailed information of the image, such as edges, textures, etc., the maximum fusion method can retain the significant part of these detailed information, making the fused image clearer and richer in details. The purpose of the maximum fusion method is to enable the fused high-frequency coefficients to retain the detailed information in the first image and the second image, while reducing noise and artifacts. Since high-frequency coefficients usually contain more noise and artifacts, by selecting larger high-frequency coefficients, the influence of these adverse factors can be suppressed to a certain extent, thereby improving the clarity and visual effect of the fused image.

[0077] S104: performing inverse wavelet transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a fused third image.

[0078] The inverse wavelet transform is the inverse process of wavelet decomposition. It uses the fused low-frequency coefficients to reconstruct the main outline and low-frequency information of the third image. Then, by gradually adding the fused high-frequency coefficients (detail components), including the detail information in the horizontal, vertical and diagonal directions, the details of the third image are improved, thus obtaining a third image with more comprehensive focus and more complete information.

[0079] In summary, the present application discloses an image fusion method, which can integrate the respective clear areas of a first image and a second image by fusing them, thereby generating a third image with a more comprehensive focus and more complete information. This fusion not only improves the utilization rate of information in the image, but also significantly improves the visual effect and practical application value of the generated image. Whether in medical imaging, remote sensing monitoring, photography art or other fields that require high-quality image fusion, the image fusion method disclosed in the embodiments of the present application has important application value.

[0080] See also Figure 2 , which is a flow chart of a second image fusion method provided in an embodiment of the present application. The method comprises:

[0081] S201: Acquire a first original image and a second original image, wherein the first original image and the second original image have different focus positions.

[0082] The first original image and the second original image are the same scene captured by different focus positions (by adjusting the focus of the camera or using different camera settings). Due to the different focus positions, the first original image and the second original image may differ in detail clarity, depth of field, etc.

[0083] S202: registering the first original image and the second original image by at least one of feature point matching, affine transformation and projective transformation to obtain a first image and a second image, wherein the first image and the second image are aligned in spatial position.

[0084] To register the first original image with the second original image by matching feature points means: first, detecting feature points, such as corner points, edge points, etc., in the first original image and the second original image respectively. Secondly, extracting the descriptors of these feature points, and matching the feature points in the first original image with the feature points in the second original image by the similarity between the descriptors. After the matching is successful, these feature point pairs can be used to estimate the transformation relationship between the first original image and the second original image, thereby registering the first original image with the second original image to obtain the first image and the second image aligned in spatial position.

[0085] Aligning the first original image with the second original image by affine transformation means: first, detecting feature points in the first original image and the second original image respectively. Secondly, extracting the descriptors of these feature points, and matching the feature points in the first original image with the feature points in the second original image by the similarity between the descriptors. After the matching is successful, the affine transformation matrix that can transform the first original image to align with the second original image is calculated based on the matched feature point pairs using the least squares method or other optimization algorithms. The affine transformation matrix contains transformation parameters such as translation, rotation, and scaling. After obtaining the affine transformation matrix, the affine transformation matrix is ​​applied to the first original image to transform it to a state aligned with the second original image in spatial position, that is, obtaining the first image and the second image aligned in spatial position.

[0086] Aligning the first original image with the second original image by projective transformation means: first, detecting feature points in the first original image and the second original image respectively. Secondly, extracting the descriptors of these feature points, and matching the feature points in the first original image with the feature points in the second original image by the similarity between the descriptors. After the match is successful, the perspective transformation matrix is ​​calculated based on the matched feature point pairs using the perspective transformation solution algorithm (such as the direct linear transformation algorithm). This perspective transformation matrix includes transformation parameters such as translation, rotation, scaling, and perspective distortion. After obtaining the perspective transformation matrix, the perspective transformation matrix is ​​applied to the first original image to transform it to a state aligned with the second original image in spatial position, that is, obtaining the first image and the second image aligned in spatial position.

[0087] S203: Perform two-dimensional discrete wavelet transform on the first image and the second image respectively to obtain a low-frequency subband, a horizontal high-frequency subband, a vertical high-frequency subband and a diagonal high-frequency subband of the first image, and a low-frequency subband, a horizontal high-frequency subband, a vertical high-frequency subband and a diagonal high-frequency subband of the second image, wherein the two-dimensional discrete wavelet transform includes filtering in the horizontal and vertical directions.

[0088] After obtaining the first image and the second image, the first image and the second image are horizontally filtered and vertically filtered in sequence to obtain a low-frequency subband, a horizontal high-frequency subband, a vertical high-frequency subband and a diagonal high-frequency subband of the first image, and a low-frequency subband, a horizontal high-frequency subband, a vertical high-frequency subband and a diagonal high-frequency subband of the second image.

[0089] Taking horizontal filtering and vertical filtering of a two-dimensional discrete image I(x,y) as an example, the obtained low-frequency subband can be shown in the following formula (1), the obtained horizontal high-frequency subband can be shown in the following formula (2), the obtained vertical high-frequency subband can be shown in the following formula (3), and the obtained diagonal high-frequency subband can be shown in the following formula (4):

[0090]

[0091]

[0092]

[0093]

[0094] Among them, A j,k (x, y) is the low-frequency subband, j is the scale, k is the position, u is the horizontal coordinate of the pixel, and v is the vertical coordinate of the pixel. is a low-pass wavelet function of scale j and position k. is the horizontal high frequency subband, is the high-pass horizontal wavelet function of scale j and position k. is the vertical high frequency subband, is the high-pass vertical wavelet function of scale j and position k. is the diagonal high frequency subband, is a high-pass diagonal wavelet function of scale j and position k.

[0095] S204: Extracting low-frequency coefficients of the first image and low-frequency coefficients of the second image from the low-frequency subband of the first image and the low-frequency subband of the second image respectively; extracting high-frequency coefficients of the first image and high-frequency coefficients of the second image from the horizontal high-frequency subband, vertical high-frequency subband and diagonal high-frequency subband of the first image, and the horizontal high-frequency subband, vertical high-frequency subband and diagonal high-frequency subband of the second image respectively.

[0096] According to the low-frequency subband and high-frequency subband of the first image and the low-frequency subband and high-frequency subband of the second image, the low-frequency coefficients A of the first image and the second image can be obtained. j,k , horizontal high frequency coefficients of the first image and the second image Vertical high frequency coefficients of the first image and the second image And the diagonal high frequency coefficients of the first image and the second image

[0097] S205: Generate a first matrix according to the low-frequency coefficients and high-frequency coefficients of the first image and the low-frequency coefficients and high-frequency coefficients of the second image.

[0098] The first matrix is ​​a non-negative matrix (NM), which contains all important information from the two images, including low-frequency information (representing the overall structure of the first image and the second image) and high-frequency information (representing the details and edges of the first image and the second image).

[0099] S206: Perform non-negative matrix decomposition on the first matrix to obtain a basis matrix and a coefficient matrix.

[0100] Non-negative Matrix Factorization (NMF) is a method of decomposing a matrix into two non-negative matrices under the condition that all elements in the matrix are non-negative. In the present application, the first matrix is ​​decomposed into two non-negative matrices: a basis matrix and a coefficient matrix. The basis matrix contains a set of basic images or features that can be combined into the original first image and the second image. The coefficient matrix contains the weights or coefficients of each basic image or feature when reconstructing the original first image and the second image.

[0101] It should be noted that the formula for performing non-negative matrix decomposition on the first matrix can be shown as the following formula (5):

[0102] V M×N =W M×R H R×N (5)

[0103] Among them, the first matrix V is an M×N matrix, the base matrix W is an M×R matrix, the coefficient matrix H is an R×N matrix, and R is the dimension of the non-negative matrix dispersion. It should be noted that, in general, R needs to satisfy the following formula (6):

[0104]

[0105] S207: Reconstruct the fused coefficients of the base matrix and the coefficient matrix to obtain fused low-frequency coefficients and fused high-frequency coefficients.

[0106] In some specific implementations, the method for reconstructing the fusion coefficients may be:

[0107] First, after extracting the low-frequency coefficient part in the coefficient matrix H, a weighted average fusion method is used to generate a first coefficient value based on the low-frequency coefficient. This weighted average fusion method can adjust the weights according to the characteristics and quality of the local image to ensure that the low-frequency part of the fused image is smooth and retains the main features.

[0108] Secondly, extract the high-frequency coefficient part in the coefficient matrix H, and use the maximum fusion method to obtain the second coefficient value. The high-frequency coefficient mainly carries the details and edge information of the image, so using the maximum method can help retain more details in the fused image.

[0109] Subsequently, the first coefficient value and the second coefficient value are multiplied by the base matrix W respectively to obtain a fused low-frequency coefficient and a fused high-frequency coefficient.

[0110] S208: Performing inverse wavelet transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a fused third image.

[0111] The fused low-frequency coefficients and high-frequency coefficients are used to perform inverse wavelet transform. This step is to convert the coefficients in the frequency domain (i.e. the fused low-frequency coefficients and high-frequency coefficients) back to the spatial domain to obtain the fused image. Among them, the inverse wavelet transform is the inverse process of wavelet decomposition, which can restore the approximate representation of the original first image and the second image in the time domain (or spatial domain).

[0112] In some specific implementations, after obtaining the fused third image, the fused third image may be subjected to at least one of edge processing, color correction and smoothing processing to obtain a processed third image. Among them, edge processing is used to enhance the edge information of the image to make the outline of the image clearer. Color correction is used to adjust the color balance and tone of the image to ensure that the color of the fused image is natural and consistent. Smoothing is used to reduce noise and detail jitter in the image to make the image look smoother and more comfortable. Therefore, the above post-processing steps are intended to enhance the visual effect of the image, reduce possible artifacts or noise, and make the fused image closer to nature and clearer.

[0113] S209: Determine whether there is a fourth image with a different focus position from the fused third image. If so, perform step S203 on the third image and the fourth image.

[0114] In some specific implementations, after obtaining the fused third image, the fusion iteration step may be continued. Specifically, the fusion iteration step includes: when there is a fourth image with a different focal position from the fused third image, performing wavelet decomposition on the third image and the fourth image to obtain low-frequency coefficients and high-frequency coefficients of the third image, as well as low-frequency coefficients and high-frequency coefficients of the fourth image; generating secondary fused low-frequency coefficients and secondary fused high-frequency coefficients based on the low-frequency coefficients and high-frequency coefficients of the third image, as well as the low-frequency coefficients and high-frequency coefficients of the fourth image; performing inverse wavelet transform on the secondary fused low-frequency coefficients and the secondary fused high-frequency coefficients to obtain a fused fifth image.

[0115] It is understandable that in the image fusion method provided in the present application, the above fusion iteration steps can be repeatedly performed until there is no image to be fused with a focus position different from that of the fused image. Thus, the flexibility and applicability of image fusion can be significantly improved, especially in scenes where multiple focus areas need to be observed simultaneously.

[0116] See also Figure 3 A schematic diagram of image fusion provided in an embodiment of the present application. It can be understood that the above steps S203-S204 are Figure 3 The process of performing wavelet decomposition on the first image and the second image to obtain the low-frequency coefficients of the first image, the low-frequency coefficients of the second image, the high-frequency coefficients of the first image and the high-frequency coefficients of the second image. The above steps S205-S206 are Figure 3 The step of performing non-negative matrix decomposition according to the low-frequency coefficients of the first image, the low-frequency coefficients of the second image, the high-frequency coefficients of the first image and the high-frequency coefficients of the second image. The above step S207 is Figure 3 The above step S208 is to reconstruct the fusion coefficients in the above step to obtain the fused low-frequency coefficients and the fused high-frequency coefficients. Figure 3 The step of obtaining the third image.

[0117] See also Figure 4 , this figure is a schematic diagram of image fusion based on a one-way aberration adjustable focus lens that simulates the human eye lens provided in an embodiment of the present application. By cleverly integrating the one-way aberration adjustable focus lens that simulates the human eye lens into the electronic camera system, the electronic camera system can continuously capture the target object at different focal lengths. This series of shooting operations generates a set of image sequences with different focal points, which are clearly focused one by one from "A" to "Z", and comprehensively cover all detail levels of the target object. Subsequently, the image fusion method provided in the embodiment of the present application is applied to perform image fusion processing on each image in the captured image sequence. Specifically, first, two pictures are taken from the captured image for the first fusion processing. For example, Figure 4The clear image with the letter "A" in focus is shown in A. Figure 4 The image shown in B is focused on the letter "C" and image fusion is performed to obtain Figure 4 The fusion effect is shown in C in the figure. Then, each image in the image sequence is subjected to image fusion processing one by one to ensure that each focus information is fully utilized and integrated. Finally, as shown in Figure 4 As shown in D in the figure, a fused image with all focus information, rich details and extremely high clarity can be obtained.

[0118] In summary, the present application discloses an image fusion method, which can integrate the respective clear areas of a first image and a second image by fusing them, thereby generating a third image with a more comprehensive focus and more complete information. This fusion not only improves the utilization rate of information in the image, but also significantly improves the visual effect and practical application value of the generated image. Whether in medical imaging, remote sensing monitoring, photography art or other fields that require high-quality image fusion, the image fusion method disclosed in the embodiments of the present application has important application value.

[0119] See also Figure 5 , which is a schematic diagram of an image fusion device provided in an embodiment of the present application. The image fusion device 500 includes: an image acquisition module 501, a wavelet decomposition module 502, a coefficient generation module 503 and an image fusion module 504;

[0120] An image acquisition module 501 is used to acquire a first image and a second image, where the focus positions of the first image and the second image are different;

[0121] A wavelet decomposition module 502 is used to perform wavelet decomposition on the first image and the second image respectively to obtain low-frequency coefficients and high-frequency coefficients of the first image, and low-frequency coefficients and high-frequency coefficients of the second image;

[0122] A coefficient generation module 503, configured to generate fused low-frequency coefficients and high-frequency coefficients according to the low-frequency coefficients and high-frequency coefficients of the first image and the low-frequency coefficients and high-frequency coefficients of the second image by non-negative matrix decomposition;

[0123] The image fusion module 504 is used to perform inverse wavelet transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a fused third image.

[0124] Optionally, the coefficient generation module 503 includes: a first generation submodule, a second generation submodule and a third generation submodule;

[0125] A first generating submodule, used for generating a first matrix according to the low-frequency coefficients and the high-frequency coefficients of the first image and the low-frequency coefficients and the high-frequency coefficients of the second image;

[0126] The second generating submodule is used for performing non-negative matrix decomposition on the first matrix to obtain a base matrix and a coefficient matrix;

[0127] The third generating submodule is used for reconstructing the fused coefficients of the base matrix and the coefficient matrix to obtain fused low-frequency coefficients and fused high-frequency coefficients.

[0128] Optionally, the third generation submodule includes: an acquisition submodule and a fusion submodule;

[0129] An acquisition submodule, used for acquiring a first coefficient value by a weighted average fusion method according to the low-frequency coefficients in the coefficient matrix;

[0130] The acquisition submodule is further used to obtain the second coefficient value through a maximum fusion method according to the high frequency coefficients in the coefficient matrix;

[0131] The fusion submodule is used to multiply the first coefficient value and the second coefficient value by the base matrix respectively to obtain a fused low-frequency coefficient and a fused high-frequency coefficient.

[0132] Optionally, the wavelet decomposition module 502 includes: a first decomposition submodule and a second decomposition submodule;

[0133] A first decomposition submodule is used to perform a two-dimensional discrete wavelet transform on the first image and the second image respectively to obtain a low frequency subband, a horizontal high frequency subband, a vertical high frequency subband and a diagonal high frequency subband of the first image, and a low frequency subband, a horizontal high frequency subband, a vertical high frequency subband and a diagonal high frequency subband of the second image, wherein the two-dimensional discrete wavelet transform includes filtering in the horizontal direction and the vertical direction;

[0134] A second decomposition submodule is used to extract low-frequency coefficients of the first image and low-frequency coefficients of the second image from the low-frequency subband of the first image and the low-frequency subband of the second image respectively;

[0135] The second decomposition submodule is also used to extract the high-frequency coefficients of the first image and the high-frequency coefficients of the second image from the horizontal high-frequency subband, vertical high-frequency subband and diagonal high-frequency subband of the first image, and the horizontal high-frequency subband, vertical high-frequency subband and diagonal high-frequency subband of the second image, respectively.

[0136] Optionally, the image acquisition module 501 includes: a first acquisition submodule and a second acquisition submodule;

[0137] A first acquisition submodule is used to acquire a first original image and a second original image, wherein the first original image and the second original image have different focal positions;

[0138] The second acquisition submodule is used to align the first original image and the second original image by at least one of feature point matching, affine transformation and projection transformation to obtain the first image and the second image, and the first image and the second image are aligned in spatial position.

[0139] Optionally, the image fusion device 500 further includes: a post-processing module; the post-processing module is used to perform at least one of edge processing, color correction and smoothing processing on the fused third image to obtain a processed third image.

[0140] In summary, the present application discloses an image fusion device, which can integrate the respective clear areas of a first image and a second image by fusing them, thereby generating a third image with a more comprehensive focus and more complete information. This fusion not only improves the utilization rate of information in the image, but also significantly improves the visual effect and practical application value of the generated image. Whether in medical imaging, remote sensing monitoring, photography art or other fields that require high-quality image fusion, the image fusion device disclosed in the embodiment of the present application has important application value.

[0141] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image fusion method, characterized in that: The method comprises: Acquire a first image and a second image, wherein the first image and the second image have different focal positions; Performing wavelet decomposition on the first image and the second image respectively to obtain low-frequency coefficients and high-frequency coefficients of the first image, and low-frequency coefficients and high-frequency coefficients of the second image; According to the low-frequency coefficients and high-frequency coefficients of the first image and the low-frequency coefficients and high-frequency coefficients of the second image, generating fused low-frequency coefficients and high-frequency coefficients by non-negative matrix decomposition; Performing inverse wavelet transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a fused third image.

2. The method according to claim 1, characterized in that The step of generating fused low-frequency coefficients and high-frequency coefficients by non-negative matrix decomposition according to the low-frequency coefficients and high-frequency coefficients of the first image and the low-frequency coefficients and high-frequency coefficients of the second image comprises: Generate a first matrix according to the low-frequency coefficients and the high-frequency coefficients of the first image and the low-frequency coefficients and the high-frequency coefficients of the second image; Performing non-negative matrix decomposition on the first matrix to obtain a basis matrix and a coefficient matrix; The fused low-frequency coefficients and fused high-frequency coefficients are obtained by reconstructing the fused coefficients of the base matrix and the coefficient matrix.

3. The method according to claim 2, characterized in that The step of reconstructing the base matrix and the coefficient matrix by fusing coefficients to generate fused low-frequency coefficients and fused high-frequency coefficients includes: According to the low-frequency coefficients in the coefficient matrix, a first coefficient value is obtained by a weighted average fusion method; According to the high frequency coefficients in the coefficient matrix, a second coefficient value is obtained by a maximum fusion method; The first coefficient value and the second coefficient value are multiplied by the base matrix respectively to obtain a fused low-frequency coefficient and a fused high-frequency coefficient.

4. The method according to claim 1, characterized in that: The method further comprises a fusion iteration step, wherein the fusion iteration step comprises: When there is a fourth image having a different focal position from the fused third image, performing wavelet decomposition on the third image and the fourth image to obtain low-frequency coefficients and high-frequency coefficients of the third image, and low-frequency coefficients and high-frequency coefficients of the fourth image; Generate a secondary fused low-frequency coefficient and a secondary fused high-frequency coefficient according to the low-frequency coefficient and the high-frequency coefficient of the third image and the low-frequency coefficient and the high-frequency coefficient of the fourth image; Performing inverse wavelet transform on the secondary fused low-frequency coefficients and the secondary fused high-frequency coefficients to obtain a fused fifth image; The fusion iteration step is repeated until there is no image to be fused whose focal position is different from that of the fused image.

5. The method according to claim 1, characterized in that The performing wavelet decomposition on the first image and the second image respectively to obtain low-frequency coefficients and high-frequency coefficients of the first image, and low-frequency coefficients and high-frequency coefficients of the second image, comprises: Performing a two-dimensional discrete wavelet transform on the first image and the second image respectively to obtain a low frequency subband, a horizontal high frequency subband, a vertical high frequency subband, and a diagonal high frequency subband of the first image, and a low frequency subband, a horizontal high frequency subband, a vertical high frequency subband, and a diagonal high frequency subband of the second image, wherein the two-dimensional discrete wavelet transform includes filtering in the horizontal direction and the vertical direction; Extracting low-frequency coefficients of the first image and low-frequency coefficients of the second image from the low-frequency subband of the first image and the low-frequency subband of the second image respectively; The high-frequency coefficients of the first image and the high-frequency coefficients of the second image are respectively extracted from the horizontal high-frequency subband, vertical high-frequency subband and diagonal high-frequency subband of the first image and the horizontal high-frequency subband, vertical high-frequency subband and diagonal high-frequency subband of the second image.

6. The method according to claim 1, characterized in that The acquiring of the first image and the second image comprises: Acquire a first original image and a second original image, wherein the first original image and the second original image have different focal positions; The first original image and the second original image are registered by at least one of feature point matching, affine transformation and projective transformation to obtain the first image and the second image, and the first image and the second image are aligned in spatial position.

7. The method according to any one of claims 1 to 6, characterized in that: After performing the inverse wavelet transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a fused third image, the method further includes: At least one of edge processing, color correction and smoothing processing is performed on the fused third image to obtain a processed third image.

8. An image fusion device, characterized in that: The device comprises: an image acquisition module, a wavelet decomposition module, a coefficient generation module and an image fusion module; The image acquisition module is used to acquire a first image and a second image, wherein the focus positions of the first image and the second image are different; The wavelet decomposition module is used to perform wavelet decomposition on the first image and the second image respectively to obtain low-frequency coefficients and high-frequency coefficients of the first image, and low-frequency coefficients and high-frequency coefficients of the second image; The coefficient generation module is used to generate fused low-frequency coefficients and high-frequency coefficients according to the low-frequency coefficients and high-frequency coefficients of the first image and the low-frequency coefficients and high-frequency coefficients of the second image through non-negative matrix decomposition; The image fusion module is used to perform inverse wavelet transform on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain a fused third image.

9. An image fusion device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the method according to any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method according to any one of claims 1 to 7 is implemented.

Images