Image fusion method and device, storage medium and electronic device

Abstract

This application discloses an image fusion method, apparatus, storage medium, and electronic device. The method includes: performing image transformations in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, wherein the low-frequency image group includes low-frequency images corresponding to each image, and the high-frequency image group includes high-frequency images corresponding to each image; dividing the images in the high-frequency image group into blocks according to a preset size to obtain multiple image block pairs corresponding to the high-frequency image group; performing image block fusion on the image blocks within each of the multiple image block pairs to obtain multiple fused image blocks; stitching the multiple fused image blocks together to obtain a high-frequency fused image; performing image fusion on the images in the low-frequency image group to obtain a low-frequency fused image; and performing an inverse image transformation on the high-frequency fused image and the low-frequency fused image to obtain a fused image of at least one image.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to an image fusion method and apparatus, a storage medium, and an electronic device. Background Technology

[0002] In related technologies, multi-sensor image fusion (e.g., visible light images, infrared images, etc.) can be widely used in target recognition, machine vision, remote sensing, and medical image processing. To address the problem of low efficiency in multi-scale transformation fusion methods due to the need to process large amounts of data, compressed sensing can be used for multi-sensor image fusion. Compressed sensing-based methods can compress data during sampling to reduce the amount of data and improve data processing efficiency.

[0003] For example, a single-layer wavelet decomposition can be performed on visible light images and infrared images; the low-frequency subband coefficients (subband coefficients are images) after wavelet decomposition can be fused to obtain fused low-frequency subband coefficients; the high-frequency subband coefficients can be globally weighted and fused to obtain fused high-frequency subband coefficients; and the fused low-frequency subband coefficients and high-frequency subband coefficients can be subjected to inverse wavelet transform to obtain the fused image.

[0004] However, globally weighted fusion of high-frequency subband coefficients involves high computational complexity and a large amount of calculation, resulting in low image fusion efficiency. Therefore, it is evident that image fusion methods in related technologies suffer from low efficiency due to the high computational complexity of the fusion calculations. Summary of the Invention

[0005] This application provides an image fusion method and apparatus, a storage medium and an electronic device, to at least solve the problem of low efficiency in image fusion methods in the related art due to the high complexity of fusion calculations.

[0006] According to one aspect of the embodiments of this application, an image fusion method is provided, comprising: performing image transformations in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, wherein the low-frequency image group includes low-frequency images corresponding to each image, and the high-frequency image group includes high-frequency images corresponding to each image; dividing the images in the high-frequency image group into blocks according to a preset size to obtain multiple image block pairs corresponding to the high-frequency image group; performing image block fusion on the image blocks in each of the multiple image block pairs to obtain multiple fused image blocks, and stitching the multiple fused image blocks together to obtain a high-frequency fused image; performing image fusion on the images in the low-frequency image group to obtain a low-frequency fused image; and performing inverse image transformation on the high-frequency fused image and the low-frequency fused image to obtain a fused image of the at least one image.

[0007] According to another aspect of the embodiments of this application, an image fusion apparatus is provided, comprising: a transformation unit, configured to perform image transformations in multiple directions on at least one image to be fused, to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, wherein the low-frequency image group includes low-frequency images corresponding to each image, and the high-frequency image group includes high-frequency images corresponding to each image; a processing unit, configured to perform block processing on the images in the high-frequency image group according to a preset size, to obtain multiple image block pairs corresponding to the high-frequency image group; an execution unit, configured to perform image block fusion on the image blocks in each of the multiple image block pairs, to obtain multiple fused image blocks, and to perform image block stitching on the multiple fused image blocks to obtain a high-frequency fused image; a fusion unit, configured to perform image fusion on the images in the low-frequency image group, to obtain a low-frequency fused image; and an inverse transformation unit, configured to perform an inverse image transformation on the high-frequency fused image and the low-frequency fused image, to obtain a fused image of the at least one image.

[0008] In an exemplary embodiment, the transformation unit includes a transformation module, configured to perform a one-layer non-downsampled contourlet transformation on the at least one image to be fused in four directions to obtain the low-frequency image group and the high-frequency image group.

[0009] In an exemplary embodiment, the execution unit includes: a first execution module, configured to multiply a measurement matrix by each image block in each image block pair to obtain a plurality of observed image block pairs, wherein the number of columns in the measurement matrix is ​​the same as the number of rows in each image block pair; a fusion module, configured to fuse the pixels of each of the plurality of observed image block pairs based on the matching degree between the region energies of the pixels to obtain a plurality of fused measurement image blocks; and a reconstruction module, configured to reconstruct each of the plurality of fused measurement image blocks using a perceptual matrix to obtain the plurality of fused image blocks, wherein the perceptual matrix is ​​obtained by multiplying the measurement matrix by a redundant dictionary matching the measurement matrix.

[0010] In an exemplary embodiment, the fusion module includes: a first execution submodule, configured to perform the following operations on each observation image block pair as a current observation image block pair to obtain the plurality of fused measurement image blocks: performing the following fusion operations on each pixel point in the current observation image block pair as a current pixel point: determining the matching degree between the regional energies of the current observation image block pair at the current pixel point to obtain a current matching degree; if the current matching degree is greater than or equal to a matching degree threshold, performing a weighted summation of the pixel values ​​of the current observation image block pair at the current pixel point to obtain the pixel value of the current pixel point in the fused measurement image block corresponding to the current observation image block pair; if the current matching degree is less than a matching degree threshold, determining the pixel value of the target observation image block at the current pixel point as the pixel value of the current pixel point in the fused measurement image block corresponding to the current observation image block pair, wherein the target observation image block is the observation image block with the largest regional energy at the current pixel point in the current observation image block pair.

[0011] In an exemplary embodiment, the apparatus further includes: a determining unit, configured to, before performing a weighted summation of the pixel values ​​of the current pixel in the current observed image block pair, determine a first weight by multiplying the ratio of a first difference to a second difference with a preset coefficient, and determine a second weight by the difference between 1 and the first weight, wherein the first difference is the difference between the current matching degree and the matching degree threshold, the second difference is the difference between 1 and the matching degree threshold, the first weight is the weighting coefficient corresponding to the observed image block with the largest pixel value of the current pixel in the current observed image block pair, and the second weight is the weighting coefficient corresponding to the observed image block with the smallest pixel value of the current pixel in the current observed image block pair.

[0012] In an exemplary embodiment, the reconstruction module includes: a second execution submodule, configured to perform the following reconstruction operations on each column of each fused measurement image block using the perception matrix to obtain the plurality of fused image blocks, wherein, during the execution of the following reconstruction operations, each column is a current column: constructing sparse coefficients matching the current column using the perception matrix; multiplying the sparse coefficients by the redundant dictionary to obtain the current column in the fused image block corresponding to each fused measurement image block.

[0013] In an exemplary embodiment, the fusion unit includes: a second execution module, configured to perform the following fusion operations on each pixel of the images in the low-frequency image group as the current pixel to obtain the pixel value of the current pixel in the low-frequency fused image: determining the pixel value variance of a group of pixels in a preset neighborhood centered on the current pixel in each image of the low-frequency image group to obtain the neighborhood variance corresponding to each image; and determining the largest neighborhood variance among the neighborhood variances corresponding to each image as the pixel value of the current pixel in the low-frequency fused image.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which is configured to execute the above-described image fusion method at runtime.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the image fusion method described above through the computer program.

[0016] In this embodiment, a method is adopted to fuse the high-frequency image blocks obtained by dividing the high-frequency image into blocks. This is achieved by performing image transformations in multiple directions on at least one image to be fused, resulting in a low-frequency image group and a high-frequency image group corresponding to the at least one image. The low-frequency image group includes low-frequency images corresponding to each image, and the high-frequency image group includes high-frequency images corresponding to each image. The images in the high-frequency image group are divided into blocks according to a preset size, resulting in multiple image block pairs corresponding to the high-frequency image group. Image block fusion is performed on the image blocks within each of the multiple image block pairs to obtain multiple fused image blocks. These fused image blocks are then stitched together to obtain a high-frequency fused image. Image fusion is performed on the images in the low-frequency image group to obtain a low-frequency fused image. Finally, an inverse image transformation is performed on the high-frequency fused image and the low-frequency fused image to obtain a fused image of at least one image. By dividing the high-frequency image into blocks, fusing the resulting high-frequency image blocks, and then stitching the fused image blocks together to obtain the high-frequency fused image, the grid effect between blocks during block-based solving can be reduced. This reduces the computational complexity of image fusion and improves the efficiency of image fusion. This solves the problem of low efficiency in image fusion methods due to high computational complexity in related technologies. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the hardware environment of an optional image fusion method according to an embodiment of this application;

[0020] Figure 2 This is a schematic flowchart of an optional image fusion method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of an optional image fusion method according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of another optional image fusion method according to an embodiment of this application;

[0023] Figure 5 This is a flowchart illustrating another optional image fusion method according to an embodiment of this application;

[0024] Figure 6 This is a structural block diagram of an optional image fusion apparatus according to an embodiment of this application;

[0025] Figure 7 This is a structural block diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] According to one aspect of the embodiments of this application, an image fusion method is provided. Optionally, in this embodiment, the above-described image fusion method can be applied to, for example... Figure 1 The hardware environment shown includes terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to terminal device or clients installed on terminal device. A database can be set up on the server or independently of the server to provide data storage services for server 104.

[0029] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may be, but is not limited to, devices such as smartphones, smart computers, and smart tablets.

[0030] The image fusion method of this embodiment can be executed by server 104, by terminal device 102, or by both server 104 and terminal device 102. Taking the execution of the image fusion method of this embodiment by terminal device 102 as an example... Figure 2 This is a flowchart illustrating an optional image fusion method according to an embodiment of this application, as shown below. Figure 2 As shown, the process of this method may include the following steps:

[0031] Step S202: Perform image transformations in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to at least one image. The low-frequency image group includes the low-frequency images corresponding to each image, and the high-frequency image group includes the high-frequency images corresponding to each image.

[0032] The image fusion method in this embodiment can be applied to at least one scenario requiring image fusion. Here, the fused one or more images can be multi-sensor images, which can be images of the same scene in different bands, or other types of images requiring image fusion. The multiple images can be images acquired by the same image acquisition component or images acquired by different image acquisition components. The aforementioned at least one image can include, but is not limited to, at least one of the following: visible light images, infrared images, and correspondingly, the aforementioned image acquisition components can be cameras, infrared sensors, etc.

[0033] In related technologies, multi-sensor image fusion methods mainly include: multi-scale transform fusion methods and compressed sensing-based methods. Among them, multi-scale transform fusion methods require processing a large amount of data and are not efficient, while compressed sensing-based methods compress data during sampling, reducing the amount of data and improving efficiency. However, they do not fully decompose the image information, resulting in high reconstruction complexity and poor high-frequency information fusion effect.

[0034] For example, in image fusion based on compressed sensing, a single-layer wavelet decomposition can be performed on the input visible light and infrared images to obtain low-frequency and high-frequency sub-band coefficients. The low-frequency sub-band coefficients are then fused, and the high-frequency sub-band coefficients are fused based on the observed coefficients. Finally, an inverse wavelet transform is performed on the fused low-frequency and high-frequency sub-band coefficients to obtain the fused image. However, this method of globally weighted fusion of the observed coefficients of the high-frequency components is computationally intensive, and the selection of fusion weights cannot be adapted to different regions, thus compromising accuracy.

[0035] To at least partially solve the above problems, in this embodiment, the high-frequency subband coefficients obtained by image transformation can be divided into blocks, and the corresponding image blocks obtained by the blocks can be fused. Since the high-frequency subband coefficients are divided into blocks, the global fusion of the image is split into the fusion of image blocks, which can reduce the amount of computation required for image fusion. At the same time, the parameters used in the fusion (e.g., fusion weights) can be applied to each region, which can improve the accuracy of image fusion.

[0036] For at least one image to be fused, such as a visible light image (Image A) and an infrared image (Image B), each image can be transformed in multiple directions to obtain a low-frequency image and a high-frequency image corresponding to each image, thus obtaining a low-frequency image group and a high-frequency image group. The dimensions of each image in the low-frequency image group and the dimensions of each image in the high-frequency image group can be the same. The number of low-frequency image groups can be one or more, for example, one, and the number of high-frequency image groups can be one or more, for example, four. The processing procedures for different low-frequency image groups are similar, and the processing procedures for different high-frequency image groups are also similar. Unless otherwise specified, in this embodiment, the method for fusing low-frequency image groups is applicable to all low-frequency image groups, and the method for fusing high-frequency image groups is applicable to all high-frequency image groups.

[0037] The aforementioned image transformation may include image decomposition, which may involve performing multiple transformations of different directions and scales on each image. The low-frequency image (i.e., low-frequency subband coefficients) and high-frequency image (i.e., high-frequency subband coefficients) obtained by performing image transformation on each image may be multiple sub-images of different directions and scales obtained by decomposing each image.

[0038] Optionally, before image fusion, to improve the quality of image fusion, the two images can be preprocessed to align scenes in different images (i.e., scene alignment). The preprocessing operations may include angle transformation operations, etc. This embodiment does not limit the image preprocessing procedure.

[0039] Step S204: Divide the images in the high-frequency image group into blocks according to the preset size to obtain multiple image block pairs corresponding to the high-frequency image group.

[0040] For a high-frequency image group, each high-frequency image group can contain two high-frequency images. For each high-frequency image, it can be divided into blocks according to a preset size to obtain multiple image block pairs corresponding to the high-frequency image group. For example, a sliding window can be used to divide the images in the high-frequency image group into blocks of a preset size, with partial overlap between adjacent blocks, thereby obtaining multiple image block pairs corresponding to the high-frequency image group.

[0041] For example, a sliding window can be used to divide a high-frequency image into n×n blocks, maintaining partial overlap between adjacent blocks. For different high-frequency images in a high-frequency image group, image blocks at the same location can be considered as a pair of image blocks. Since each high-frequency image can be divided into multiple image blocks, multiple pairs of image blocks can be obtained at the end of the segmentation process.

[0042] Step S206: Perform image block fusion on the image blocks in each of the multiple image block pairs to obtain multiple fused image blocks, and then stitch the multiple fused image blocks together to obtain a high-frequency fused image.

[0043] After obtaining multiple image block pairs, the terminal device can perform image block fusion on each of the multiple image block pairs. The fusion method for each image block pair can be similar to the method for fusing the entire high-frequency image. For example, based on compressed sensing, each image block pair can be fused separately to obtain a fused image block corresponding to each image block pair, thereby obtaining multiple fused image blocks. Here, the multiple image block pairs correspond one-to-one with the multiple fused image blocks; that is, one image block pair is fused into one fused image block. Optionally, to improve the efficiency of image block fusion, multiple threads can be used to perform image block fusion on different image block pairs, and each thread can be used to perform image block fusion on at least one image block pair.

[0044] After obtaining multiple fused image blocks, the images can be stitched together according to their positions in the high-frequency image to obtain a high-frequency fused image. Alternatively, the images can be stitched together according to their positional relationships to obtain a high-frequency fused image. During the stitching process, for pixels that overlap between adjacent images, the pixel value of that pixel in any image block or the average of the pixel values ​​of all pixels can be taken as the pixel value of that overlapping pixel in the high-frequency fused image.

[0045] For example, after obtaining high-frequency fused image blocks (i.e., fused image blocks), the image blocks can be stitched together, and the average value of the overlapping parts can be taken to obtain a high-frequency fused image.

[0046] Step S208: Perform image fusion on the images in the low-frequency image group to obtain a low-frequency fused image.

[0047] The terminal device can perform image fusion on low-frequency image groups to obtain a low-frequency fused image. The method for fusing low-frequency image groups can be: using the mean method (i.e., global mean) to perform image fusion on low-frequency image groups, or other image fusion methods can be used. In this embodiment, the method for fusing low-frequency images is not limited.

[0048] Step S210: Perform an inverse image transformation on the high-frequency fused image and the low-frequency fused image to obtain a fused image of at least one image.

[0049] After obtaining the high-frequency fused image and the low-frequency fused image, the terminal device can perform an inverse image transformation on the high-frequency fused image and the low-frequency fused image corresponding to the aforementioned image transformation, thereby obtaining a fused image of at least one image.

[0050] For example, by performing an inverse transform on the obtained low-frequency fused image and high-frequency fused image, a fused image of visible light and infrared images can be obtained.

[0051] Through steps S202 to S210, image transformations are performed in multiple directions on at least one image to be fused, resulting in low-frequency image groups and high-frequency image groups corresponding to at least one image. The low-frequency image groups include low-frequency images corresponding to each image, and the high-frequency image groups include high-frequency images corresponding to each image. The images in the high-frequency image groups are divided into blocks according to preset sizes, resulting in multiple image block pairs corresponding to the high-frequency image groups. Image block fusion is performed on the image blocks within each of the multiple image block pairs, resulting in multiple fused image blocks. These fused image blocks are then stitched together to obtain a high-frequency fused image. Image fusion is performed on the images in the low-frequency image groups to obtain a low-frequency fused image. Inverse image transformations are performed on the high-frequency and low-frequency fused images to obtain a fused image of at least one image. This process solves the problem of low efficiency in image fusion methods due to high computational complexity in related technologies, thus improving the efficiency of image fusion.

[0052] In one exemplary embodiment, image transformations are performed on at least one image to be fused in multiple directions to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, including:

[0053] S11, perform a layer of non-downsampled contour wave transform in four directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group.

[0054] There are various image transformations that can be performed on the images to be fused, such as wavelet transform. To improve the drawback of contour waves being prone to changes during translation and to facilitate subsequent image processing, NSCT (Nonsub Sampled Contourlet Transform) or similar translation-invariant image transformation methods (e.g., curve wavelet transform, contour wavelet transform, etc.) can be used to perform image transformations on each image to be fused separately.

[0055] In this embodiment, the two images to be fused can be subjected to NSCT transforms in multiple directions to obtain low-frequency image groups and high-frequency image groups. The number of high-frequency image groups obtained will vary depending on the number of NSCT transform layers performed. Here, compared to other image transforms, the NSCT transform, due to its upsampling operation, can improve the defect of easy changes during contour wave translation, effectively highlighting detailed information and improving the quality of the fused image.

[0056] For example, visible light and infrared images can be decomposed into multiple sub-images of different directions and scales using NSCT. Visible light and infrared images can be decomposed into a low-frequency image and multiple high-frequency images, respectively.

[0057] To reduce the complexity and improve the efficiency of image fusion, a four-way NSCT transform can be performed on at least one image to be fused. Each image undergoing a four-way NSCT transform is divided into a low-frequency image and four high-frequency images, resulting in a low-frequency image group and four high-frequency image groups. Here, each high-frequency image group contains a high-frequency image obtained by performing an NSCT transform in the same direction on each of at least one image.

[0058] For example, such as Figure 3 As shown, a four-directional NSCT transform can be performed on the visible light image and infrared image to be fused separately. After the image transformation, each image can obtain 5 sub-images, including 1 low-frequency image and 4 high-frequency images, that is, a total of 1 low-frequency image group and 4 high-frequency image groups are obtained. In the decomposition and reconstruction process of NSCT, there is no downsampling process. It inherits the multidirectional and anisotropic image representation of traditional contourlet transform, and has translation invariance that traditional contourlet transform does not have.

[0059] In this embodiment, by performing a one-layer four-directional NSCT transformation on the images to be fused, the complexity of image fusion can be reduced and the efficiency of image fusion can be improved.

[0060] In one exemplary embodiment, image block fusion is performed on the image blocks within each of a plurality of image block pairs to obtain a plurality of fused image blocks, including:

[0061] S21, Multiply the measurement matrix by the image blocks in each image block pair to obtain multiple observation image block pairs, wherein the number of columns in the measurement matrix is ​​the same as the number of rows in each image block pair;

[0062] S22, based on the matching degree between the regional energies of pixels, the pixels of each of the multiple observation image block pairs are fused to obtain multiple fused measurement image blocks;

[0063] S23, each fusion measurement image block in multiple fusion measurement image blocks is reconstructed using a perceptual matrix to obtain multiple fusion image blocks, wherein the perceptual matrix is ​​obtained by multiplying the measurement matrix and a redundant dictionary that matches the measurement matrix.

[0064] In this embodiment, image block fusion can be performed on each pair of image blocks based on the observation coefficients. Correspondingly, the parameters used for image block fusion may include, but are not limited to, a measurement matrix, a redundant dictionary (matched with the measurement matrix), and a sensing matrix. The measurement matrix and the redundant dictionary can be randomly constructed or obtained by training using training samples based on a set target, and the sensing matrix is obtained by multiplying the measurement matrix and the redundant dictionary. In this embodiment, the construction method of the measurement matrix and the redundant dictionary is not limited.

[0065] For example, a random Bernoulli measurement matrix Φ and a DCT (Discrete Cosine Transform) redundant dictionary Ψ can be constructed. The dimension of the measurement matrix is m×n, where m < n, and the dimension of the redundant dictionary is n×4n. Multiplying the two results in a sensing matrix D.

[0066] For each pair of image blocks, the number of columns of the measurement matrix is the same as the number of rows of the image blocks in each pair of image blocks. The measurement matrix can be multiplied with the image blocks in each pair of image blocks respectively to obtain corresponding observed image block pairs, thus obtaining multiple observed image block pairs. For example, the measurement matrix can be multiplied with the high-frequency image blocks respectively to obtain the high-frequency observed image blocks of the visible light image and the infrared image.

[0067] In this embodiment, whether it is a high-frequency image (image block) or a low-frequency image, the image size (the number of pixel points included) of the fused image obtained by image fusion is the same as the image size of the image before fusion. That is, the image obtained by fusing two images with a size of C×D is still an image of C×D. For the current observed image block pair (which can be any observed image block pair), the pixel values of each pixel point in the current observed image block pair can be fused respectively based on the matching degree between the regional energies of the pixel points to obtain corresponding fused measurement image blocks. Multiple observed image block pairs can obtain multiple fused measurement image blocks.

[0068] When performing the current pixel point of the current observed image block pair (any pixel point in the current observed image block pair), the regional energy within the preset area corresponding to the current pixel point in each observed image block of the current observed image block pair can be determined respectively; based on the matching degree between the determined regional energies (i.e., the regional energy matching degree), the pixel value of the current pixel point in the current observed image block pair is fused to obtain the pixel value of the current pixel point in the fused measurement image block.

[0069] Here, when fusing the pixel values of the current pixel point based on the regional energy matching degree, the weighting coefficients of each observed image block can be determined based on the regional energy matching degree, and the pixel values of each observed image block at the current pixel point are weighted and fused based on the determined weighting coefficients, so as to obtain the pixel value of the current pixel point in the fused measurement image block.

[0070] For example, high-frequency coefficient fusion rules can be used to fuse high-frequency observation image patches. When fusing high-frequency observation image patches, the regional energy E of high-frequency observation image patches HA and HB at pixel (x,y) can be calculated based on formula (1). A (x,y),E B (x,y):

[0071]

[0072] Where H is the image patch, L×W is the energy region size (which can be set to 3×3), and ω is the weighting matrix (which can be...).

[0073] For each fused measurement image block, a perceptual matrix can be used to reconstruct the image block based on compressed sensing, resulting in multiple fused image blocks. The method of image block reconstruction using the perceptual matrix can refer to relevant technologies, and is not limited to this in this embodiment.

[0074] In this embodiment, the observation image block is obtained by multiplying the observation matrix with the high-frequency image block, and then fusion is performed using the region energy matching method. The pixel value of the pixel is determined by the energy characteristics of the image in the surrounding region centered on the pixel. The determined pixel value contains the high-frequency features of the image in the region, which can improve the fusion effect, retain more details, and reduce noise.

[0075] In one exemplary embodiment, pixels in each of a plurality of observed image patch pairs are fused based on the matching degree between region energies of pixels to obtain a plurality of fused measurement image patches, including:

[0076] S31, perform the following operations on each observation image block pair as the current observation image block pair to obtain multiple fused measurement image blocks:

[0077] Perform the following fusion operation on each pixel in the currently observed image patch pair as the current pixel:

[0078] Determine the matching degree between the regional energies of the current observed image patch pair at the current pixel point to obtain the current matching degree;

[0079] If the current matching degree is greater than or equal to the matching degree threshold, the pixel values ​​of the current pixel point in the current observed image block pair are weighted and summed to obtain the pixel value of the current pixel point in the fused measurement image block corresponding to the current observed image block pair.

[0080] If the current matching degree is less than the matching degree threshold, the pixel value of the target observation image block at the current pixel point is determined as the pixel value of the current pixel point in the fused measurement image block corresponding to the current observation image block pair. The target observation image block is the observation image block with the largest regional energy at the current pixel point in the current observation image block pair.

[0081] In this embodiment, the same processing operation can be used to perform observation image block fusion on each of multiple observation image block pairs. Simultaneously, when performing image block fusion on an observation image block pair, the same fusion operation can be used to determine the fused pixel value of each pixel. Here, the currently processed pixel is taken as the current pixel, and the observation image block pair to which the current pixel belongs is taken as the current observation image block pair, and the following fusion operations are performed respectively to obtain the pixel value of the current pixel in the fused measurement image block after the current observation image block pair is fused:

[0082] Step 1: Determine the matching degree between the regional energies of the current observed image patch pair at the current pixel point to obtain the current matching degree.

[0083] For example, the regional energy matching degree S(x,y) of each pair of high-frequency observation image blocks HA and HB can be calculated based on formula (2):

[0084]

[0085] Based on formula (2), the regional energy matching degree of each pair of high-frequency observation image blocks HA and HB at each pixel can be determined.

[0086] Step 2: If the current matching degree is greater than or equal to the matching degree threshold, perform a weighted summation of the pixel values ​​of the current pixel in the current observed image block pair to obtain the pixel value of the current pixel in the fused measurement image block corresponding to the current observed image block pair.

[0087] If the current matching degree is greater than or equal to the matching degree threshold, the pixel values ​​at the current pixel point of the current observed image block pair can be weighted and summed to obtain the pixel value of the current pixel point in the fused measurement image block corresponding to the current observed image block pair. Here, weighting and summing the pixel values ​​at the current pixel point of the current observed image block pair can be done by weighting and summing the pixel values ​​at the current pixel point of each observed image block in the current observed image block pair. The weighting coefficients corresponding to different observed image blocks can be determined based on the current matching degree or can be preset; this embodiment does not limit this.

[0088] For example, when fusing observed image patches, an energy matching threshold δ can be set, where 0.5 ≤ δ ≤ 1. When the energy matching degree is greater than or equal to this threshold, it indicates that the similarity of this region is high, and weighted fusion can be performed.

[0089] Step 3: If the current matching degree is less than the matching degree threshold, determine the pixel value of the target observation image block at the current pixel point as the pixel value of the current pixel point in the fused measurement image block corresponding to the current observation image block pair. The target observation image block is the observation image block with the largest regional energy at the current pixel point in the current observation image block pair.

[0090] If the current matching degree is less than the matching degree threshold, the observation image block with the highest regional energy at the current pixel in the current observation image block pair can be identified to obtain the target observation image block. For example, the regional energy of each observation image block in the current observation image block pair at the current pixel can be compared to determine the observation image block corresponding to the highest regional energy, thus obtaining the target observation image block. The pixel value of the target observation image block at the current pixel is then determined as the pixel value of the current pixel in the fused measurement image block corresponding to the current observation image block pair. At this point, the weighting coefficient corresponding to the target observation image block can be considered to be 1, and the weighting coefficients corresponding to the other observation image blocks can be considered to be 0.

[0091] For example, if the energy matching degree is lower than the threshold, it means that the similarity of this region is low. The region with high energy is used as the fusion result. The calculation formula is as follows:

[0092]

[0093] Where HC(x,y) is the pixel value at pixel (x,y) in the fused measurement image block.

[0094] In this embodiment, the fusion of measurement image patches based on regional energy similarity and energy matching thresholds can reduce the complexity of fusion calculations while ensuring the fusion effect.

[0095] In one exemplary embodiment, before performing a weighted summation of the pixel values ​​at the current pixel point for the currently observed image patch, the method further includes:

[0096] S41, the product of the ratio of the first difference and the second difference and the preset coefficient is determined as the first weight, and the difference between 1 and the first weight is determined as the second weight. The first difference is the difference between the current matching degree and the matching degree threshold, the second difference is the difference between 1 and the matching degree threshold, the first weight is the weighting coefficient corresponding to the observation image block with the largest pixel value of the current pixel in the current observation image block pair, and the second weight is the weighting coefficient corresponding to the observation image block with the smallest pixel value of the current pixel in the current observation image block pair.

[0097] Before performing a weighted summation of the pixel values ​​at the current pixel point in the current observation image patch pair, weighting coefficients corresponding to each observation image patch in the current observation image patch pair can be determined separately. The weighting coefficients for different observation image patches can be pre-set; for example, observation image patches with high region energy have a larger weighting coefficient, while observation image patches with low region energy have a smaller weighting coefficient. To ensure image fusion effectiveness, the weighting coefficients for each observation image patch can be determined based on the current matching degree and the matching degree threshold.

[0098] In this embodiment, the product of the difference between the current matching degree and the matching degree threshold, the ratio of 1 to the difference between 1 and the matching degree threshold, and a preset coefficient can be used to determine the weighting coefficient corresponding to the observation image block with the largest pixel value among the current pixels in the current observation image block pair; the difference between 1 and the first weight can be used to determine the weighting coefficient corresponding to the other observation image block in the current observation image block pair. The preset coefficient here can be a preset value, for example, 0.5.

[0099] For example, when weighted fusion of observed image patches can be performed based on formula (4):

[0100]

[0101] Where HC(x,y) represents the fused observation image, and w max and w min To integrate the weights, w max and w min The calculation formula is shown in formula (5):

[0102]

[0103] In this embodiment, the weighting coefficients corresponding to each observed image block are determined based on the current matching degree and the matching degree threshold, which can ensure the image fusion effect and improve the quality of the fused image.

[0104] In one exemplary embodiment, a perceptual matrix is ​​used to reconstruct each of a plurality of fused measurement image blocks to obtain a plurality of fused image blocks, including:

[0105] S51, using the perception matrix, perform the following reconstruction operation on each column of each fused measurement image block to obtain multiple fused image blocks, wherein each column is the current column during the following reconstruction operation:

[0106] Construct sparse coefficients that match the current column using the perceptual matrix;

[0107] Multiply the sparse coefficients by the redundant dictionary to obtain the current column in the fused image block corresponding to each fused measurement image block.

[0108] In this embodiment, when reconstructing each fused measurement image block, a perceptual matrix can be used to perform a reconstruction operation on each column of each fused measurement image block, thereby reconstructing the corresponding column of the fused image block, and thus obtaining the fused image block corresponding to each fused measurement image block. Multiple fused measurement image blocks can be reconstructed into multiple fused image blocks, and there is a one-to-one correspondence between them.

[0109] When reconstructing the current column to be reconstructed (which can be any column in any fused measurement image patch), the following reconstruction operations can be performed using it as the current column:

[0110] Step 1: Construct sparse coefficients that match the current column using the perceptual matrix.

[0111] For the current column, a sparse coefficient matching the current column can be constructed using the reconstruction algorithm in compressed sensing with the sensing matrix. Here, the reconstruction algorithm used can be the OMP (Orthogonal Matching Pursuit) algorithm or other reconstruction algorithms. When reconstructing the column, a set of atoms that best match the current column can be found from the sensing matrix to construct a sparse approximation and iteratively obtain a sparse coefficient.

[0112] For example, the OMP algorithm can be used to reconstruct the fused high-frequency image patch from the fused high-frequency measurement image patch HC. When reconstructing the fused high-frequency image patch, a set of atoms that best matches each column of the fused high-frequency observation image patch can be found from the perceptual matrix to construct a sparse approximation, and a sparse coefficient can be obtained iteratively. The process of obtaining the sparse coefficient can include:

[0113] 1) Let the first column of the image patch be x, initialize the residual r0 to x, and the index set A be empty.

[0114] 2) Find the perception matrix D and the residual r k The subscript of the column with the largest inner product is denoted as . d i For each atom of the perception matrix, N is the dimension of the perception matrix.

[0115] 3) Add the index of the most relevant element found to the index set, i.e., A k+1 =A k ∪{λ k+1}

[0116] 4) Update the residuals, that is,

[0117] 5) The iteration ends when the residual is less than the threshold, and the resulting sparse coefficients are:

[0118] Multiplying the sparse coefficients by the redundant dictionary yields the current column in the fused image block corresponding to each fused measurement image block.

[0119] Step 2: Multiply the sparse coefficients by the redundant dictionary to obtain the current column in the fused image block corresponding to each fused measurement image block.

[0120] For example, after obtaining the sparse coefficients, the sparse coefficients can be multiplied by the redundant dictionary (e.g., the aforementioned DWT dictionary Ψ) to obtain a column of the reconstructed image. The above operation is repeated for each column of the image block to obtain the reconstructed high-frequency fused image block.

[0121] In this embodiment, the ease of image patch reconstruction can be improved by using a perceptual matrix to construct and fuse a column of matching sparse coefficients in a measurement image patch, and then performing column reconstruction based on the coefficients and a redundant dictionary.

[0122] In one exemplary embodiment, image fusion is performed on images in a low-frequency image group to obtain a low-frequency fused image, including:

[0123] S61, taking each pixel of the image in the low-frequency image group as the current pixel, perform the following fusion operation to obtain the pixel value of the current pixel of the low-frequency fused image:

[0124] S62, determine the pixel value variance of a set of pixels in a preset neighborhood centered on the current pixel in each image of the low-frequency image group, and obtain the neighborhood variance corresponding to each image;

[0125] S63, determine the maximum neighborhood variance among the neighborhood variances corresponding to each image as the pixel value of the current pixel in the low-frequency fused image.

[0126] While using the global mean to fuse low-frequency image groups is a simple and convenient method, it can lead to the loss of some information, resulting in inaccurate contour information in the fused image. In this embodiment, the low-frequency components (i.e., low-frequency images) can be fused by taking the maximum value of the neighborhood variance. The fused low-frequency components are obtained by using variance to characterize the amount of image information within a region (i.e., the variance reflects the dispersion of image grayscale values; the larger the variance, the more information). By comparing these variances, the pixel values ​​of each point are influenced by the image features of the adjacent regions, thus preserving the main content and contours of the image to a greater extent.

[0127] For low-frequency images, the values ​​obtained by fusing pixels at each pixel location can be determined separately and used as the pixel values ​​of the corresponding pixel locations in the fused low-frequency image. For each pixel in the low-frequency image group (e.g., the pixel at position a×b), the fusion operation can be performed on each pixel by taking the maximum value of the neighborhood variance, thus obtaining the pixel value of the current pixel (e.g., the pixel at position a×b) in the low-frequency fused image. Performing the fusion operation on the current pixel can include the following operations:

[0128] Operation 1: Determine the variance of pixel values ​​of a set of pixels in a preset neighborhood centered on the current pixel in each low-frequency image, and obtain the neighborhood variance corresponding to each low-frequency image.

[0129] For example, the neighborhood variance maximum rule is used to fuse low-frequency images. For any image in the visible light or infrared image, a pixel with coordinates (x, y) can be taken, and its variance in a neighborhood of size M can be calculated using formula (6):

[0130]

[0131] Where L(x,y) is the low-frequency image, is the mean of the low-frequency images within the neighborhood M.

[0132] Operation 2 involves determining the maximum neighborhood variance among the neighborhood variances corresponding to each low-frequency image as the pixel value of the current pixel in the low-frequency fused image.

[0133] After obtaining the neighborhood variance corresponding to each low-frequency image (corresponding to the current pixel), the largest neighborhood variance among the neighborhood variances corresponding to each low-frequency image can be selected and determined as the pixel value of the current pixel in the low-frequency fused image. Alternatively, the pixel value of the low-frequency image corresponding to the largest neighborhood variance at the current pixel can be determined as the pixel value of the current pixel in the low-frequency fused image.

[0134] For example, for a single pixel, the pixel value of the visible light image and the infrared image with the larger variance in the neighborhood of that pixel can be selected as the pixel value of the low-frequency fused image.

[0135] In this embodiment, by fusing low-frequency images by taking the maximum value of the neighborhood variance, the main content and outline of the image can be preserved to a greater extent, thereby improving the accuracy of the fused image.

[0136] The image fusion method in this application embodiment is explained below with reference to an optional example. In this optional example, the two images are a visible light image and an infrared image, respectively. The image transformation is NSCT transform, the inverse transform is NSCT inverse transform, and the image patch reconstruction uses the OMP algorithm.

[0137] This optional example provides an image fusion method based on NSCT transform and compressed sensing, such as Figure 4 and Figure 5 As shown, the process of the image fusion method in this optional example may include the following steps:

[0138] In step S502, a four-way NSCT transformation is performed on the visible light image and the infrared image respectively to obtain 5 sub-images, including 1 low-frequency image and 4 high-frequency images.

[0139] Step S504: Divide the high-frequency image into blocks (maintain pixel overlap between adjacent image blocks, and take the average value of the overlapping part during reconstruction) to obtain high-frequency image blocks. Multiply the observation matrix with the high-frequency image blocks to obtain observation image blocks (i.e., high-frequency observation image blocks).

[0140] Here, high-frequency components are divided into blocks. This image segmentation is actually a global constraint, which reduces the grid effect between blocks when solving the block by block.

[0141] Step S506: The observed image blocks are fused based on the regional energy matching degree to obtain fused observed image blocks. The fused observed image blocks are reconstructed based on compressed sensing to obtain reconstructed fused image blocks (high-frequency fused image blocks). The fused image blocks are stitched together to obtain a high-frequency fused image.

[0142] Here, the high-frequency image is fused using the rules of region energy matching. The principle is as follows: the high-frequency components are divided into blocks, sparsely represented and observed to obtain the observation coefficients in the compressed domain. The observation coefficients of each block are fused using the rules of region energy matching, and the fused high-frequency image is obtained through the reconstruction algorithm.

[0143] Step S508: Based on the maximum value of the neighborhood variance, the low-frequency images are fused to obtain a low-frequency fused image.

[0144] Step S510: Perform NSCT inverse transform on the obtained fused low-frequency image and fused high-frequency image (i.e., each component after fusion) to obtain the image after fusing the visible light image and the infrared image.

[0145] This optional example allows for the full extraction of information from images at different scales and in different directions. More details are preserved after high-frequency components are fused, and computational complexity is reduced through block processing. At the same time, the average value of overlapping parts is taken to avoid grid effects. For low-frequency components, the main content and outline of the image are preserved to a greater extent.

[0146] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0148] According to another aspect of the embodiments of this application, an image fusion apparatus for implementing the above-described image fusion method is also provided. Figure 6 This is a structural block diagram of an optional image fusion apparatus according to an embodiment of this application, such as... Figure 6 As shown, the device may include:

[0149] The transformation unit 602 is used to perform image transformations in multiple directions on at least one image to be fused, to obtain a low-frequency image group and a high-frequency image group corresponding to at least one image; wherein, the low-frequency image group includes the low-frequency image corresponding to each image, and the high-frequency image group includes the high-frequency image corresponding to each image;

[0150] The processing unit 604, connected to the transformation unit 602, is used to divide the images in the high-frequency image group into blocks according to a preset size to obtain multiple image block pairs corresponding to the high-frequency image group.

[0151] The execution unit 606 is connected to the processing unit 604 and is used to perform image block fusion on the image blocks in each of the multiple image block pairs to obtain multiple fused image blocks, and to stitch the multiple fused image blocks together to obtain a high-frequency fused image.

[0152] The fusion unit 608 is connected to the execution unit 606 and is used to perform image fusion on the images in the low-frequency image group to obtain a low-frequency fused image.

[0153] The inverse transformation unit 610, connected to the fusion unit 608, is used to perform an inverse image transformation on the high-frequency fusion image and the low-frequency fusion image to obtain a fusion image of at least one image.

[0154] It should be noted that the transformation unit 602 in this embodiment can be used to execute the above step S202, the processing unit 604 in this embodiment can be used to execute the above step S204, the execution unit 606 in this embodiment can be used to execute the above step S206, the fusion unit 608 can be used to execute the above step S208, and the inverse transformation unit 610 can be used to execute the above step S210.

[0155] Through the above modules, image transformations are performed in multiple directions on at least one image to be fused, resulting in low-frequency image groups and high-frequency image groups corresponding to at least one image. The low-frequency image groups include low-frequency images corresponding to each image, and the high-frequency image groups include high-frequency images corresponding to each image. The images in the high-frequency image groups are divided into blocks according to preset sizes, resulting in multiple image block pairs corresponding to the high-frequency image groups. Image block fusion is performed on the image blocks within each of the multiple image block pairs to obtain multiple fused image blocks. These fused image blocks are then stitched together to obtain a high-frequency fused image. Image fusion is performed on the images in the low-frequency image groups to obtain a low-frequency fused image. Finally, inverse image transformations are performed on the high-frequency and low-frequency fused images to obtain a fused image of at least one image. This approach solves the problem of low efficiency in image fusion methods due to high computational complexity in related technologies, thus improving the efficiency of image fusion.

[0156] In one exemplary embodiment, the transformation unit includes:

[0157] The transformation module is used to perform a one-layer non-downsampled contour wave transformation in four directions on at least one image to be fused, to obtain a low-frequency image group and a high-frequency image group.

[0158] In one exemplary embodiment, the execution unit includes:

[0159] The first execution module is used to multiply the measurement matrix by the image blocks in each image block pair to obtain multiple observation image block pairs, wherein the number of columns in the measurement matrix is ​​the same as the number of rows in each image block pair;

[0160] The fusion module is used to fuse the pixels of each of the multiple observation image block pairs based on the matching degree between the regional energies of the pixels, so as to obtain multiple fused measurement image blocks;

[0161] The reconstruction module is used to reconstruct each of the multiple fused measurement image blocks using a perceptual matrix to obtain multiple fused image blocks. The perceptual matrix is ​​obtained by multiplying the measurement matrix and a redundant dictionary that matches the measurement matrix.

[0162] In one exemplary embodiment, the fusion module includes:

[0163] The first execution submodule is used to perform the following operations on each observation image block pair as the current observation image block pair to obtain multiple fused measurement image blocks: Each pixel in the current observation image block pair is used as the current pixel to perform the following fusion operations:

[0164] Determine the matching degree between the regional energies of the current observed image patch pair at the current pixel point to obtain the current matching degree;

[0165] If the current matching degree is greater than or equal to the matching degree threshold, the pixel values ​​of the current pixel point in the current observed image block pair are weighted and summed to obtain the pixel value of the current pixel point in the fused measurement image block corresponding to the current observed image block pair.

[0166] If the current matching degree is less than the matching degree threshold, the pixel value of the target observation image block at the current pixel point is determined as the pixel value of the current pixel point in the fused measurement image block corresponding to the current observation image block pair. The target observation image block is the observation image block with the largest regional energy at the current pixel point in the current observation image block pair.

[0167] In one exemplary embodiment, the above-described apparatus further includes:

[0168] The determining unit is used to determine the first weight by multiplying the ratio of the first difference and the second difference with a preset coefficient before performing a weighted summation of the pixel values ​​of the current pixel in the current observation image block pair, and to determine the second weight by the difference between 1 and the first weight. The first difference is the difference between the current matching degree and the matching degree threshold, the second difference is the difference between 1 and the matching degree threshold, the first weight is the weighting coefficient corresponding to the observation image block with the largest pixel value of the current pixel in the current observation image block pair, and the second weight is the weighting coefficient corresponding to the observation image block with the smallest pixel value of the current pixel in the current observation image block pair.

[0169] In one exemplary embodiment, the reconfiguration module includes:

[0170] The second execution submodule is used to perform the following reconstruction operation on each column of each fused measurement image block using the perception matrix to obtain multiple fused image blocks, wherein each column is the current column during the following reconstruction operation:

[0171] Construct sparse coefficients matching the current column using the perceptual matrix; multiply the sparse coefficients by the redundant dictionary.

[0172] Obtain the current column in the fused image block corresponding to each fused measurement image block.

[0173] In one exemplary embodiment, the fusion unit includes:

[0174] The second execution module is used to perform the following fusion operation on each pixel of the image in the low-frequency image group as the current pixel to obtain the pixel value of the current pixel of the low-frequency fused image:

[0175] Determine the variance of pixel values ​​of a set of pixels in a preset neighborhood centered on the current pixel in each image of the low-frequency image group, and obtain the neighborhood variance corresponding to each image.

[0176] The largest neighborhood variance among the neighborhood variances corresponding to each image is determined as the pixel value of the current pixel in the low-frequency fused image.

[0177] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown can be implemented through software or hardware, and the hardware environment includes the network environment.

[0178] According to another aspect of the embodiments of this application, a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to execute program code for any of the image fusion methods described above in the embodiments of this application.

[0179] Optionally, in this embodiment, the storage medium may be located on at least one of the network devices in the network shown in the above embodiment.

[0180] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

[0181] S1, perform image transformations in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to at least one image, wherein the low-frequency image group includes the low-frequency images corresponding to each image, and the high-frequency image group includes the high-frequency images corresponding to each image.

[0182] S2, divide the images in the high-frequency image group into blocks according to the preset size to obtain multiple image block pairs corresponding to the high-frequency image group;

[0183] S3, perform image block fusion on the image blocks in each of the multiple image block pairs to obtain multiple fused image blocks, and then stitch the multiple fused image blocks together to obtain a high-frequency fused image;

[0184] S4, perform image fusion on the images in the low-frequency image group to obtain a low-frequency fused image;

[0185] S5, perform inverse image transformation on the high-frequency fused image and the low-frequency fused image to obtain a fused image of at least one image.

[0186] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated in this embodiment.

[0187] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0188] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described image fusion method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0189] Figure 7 This is a structural block diagram of an optional electronic device according to an embodiment of this application, such as... Figure 7As shown, it includes a processor 702, a communication interface 704, a memory 706, and a communication bus 708. The processor 702, communication interface 704, and memory 706 communicate with each other via the communication bus 708.

[0190] Memory 706 is used to store computer programs;

[0191] When processor 702 executes a computer program stored in memory 706, it performs the following steps:

[0192] S1, perform image transformations in multiple directions on at least one image to be fused to obtain a low-frequency image group and a high-frequency image group corresponding to at least one image, wherein the low-frequency image group includes the low-frequency images corresponding to each image, and the high-frequency image group includes the high-frequency images corresponding to each image.

[0193] S2, divide the images in the high-frequency image group into blocks according to the preset size to obtain multiple image block pairs corresponding to the high-frequency image group;

[0194] S3, perform image block fusion on the image blocks in each of the multiple image block pairs to obtain multiple fused image blocks, and then stitch the multiple fused image blocks together to obtain a high-frequency fused image;

[0195] S4, perform image fusion on the images in the low-frequency image group to obtain a low-frequency fused image;

[0196] S5, perform inverse image transformation on the high-frequency fused image and the low-frequency fused image to obtain a fused image of at least one image.

[0197] Optionally, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic device and other devices.

[0198] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0199] As an example, the memory 706 described above may include, but is not limited to, the transformation unit 602, processing unit 604, execution unit 606, fusion unit 608, and inverse transformation unit 610 of the image fusion apparatus described above. Furthermore, it may include, but is not limited to, other module units of the image fusion apparatus described above, which will not be elaborated upon in this example.

[0200] The processors mentioned above can be general-purpose processors, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; they can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0201] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0202] Those skilled in the art will understand that Figure 7 The structure shown is for illustrative purposes only. The device implementing the above image fusion method can be a terminal device, such as a smartphone (e.g., Android phone, iOS phone), tablet computer, PDA, mobile Internet Device (MID), PAD, etc. Figure 7 This does not limit the structure of the aforementioned electronic device. For example, the electronic device may also include components that are more... Figure 7 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 7 The different configurations shown.

[0203] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0204] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0205] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0206] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0208] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the solution provided in this embodiment, depending on actual needs.

[0209] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or at least two units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0210] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image fusion method, characterized in that, include: At least one image to be fused is subjected to image transformation in multiple directions to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, wherein the low-frequency image group includes the low-frequency image corresponding to each image, and the high-frequency image group includes the high-frequency image corresponding to each image. The images in the high-frequency image group are divided into blocks according to a preset size to obtain multiple image block pairs corresponding to the high-frequency image group; Image block fusion is performed on the image blocks within each of the multiple image block pairs to obtain multiple fused image blocks, and the multiple fused image blocks are then stitched together to obtain a high-frequency fused image; The images in the low-frequency image group are fused to obtain a low-frequency fused image; Perform an inverse image transform on the high-frequency fused image and the low-frequency fused image to obtain a fused image of the at least one image; The step of fusing image blocks within each of the plurality of image block pairs to obtain a plurality of fused image blocks includes: multiplying a measurement matrix by each image block in each image block pair to obtain a plurality of observation image block pairs, wherein the number of columns in the measurement matrix is ​​the same as the number of rows in each image block pair; fusing the pixels of each observation image block pair based on the matching degree between the region energies of the pixels to obtain a plurality of fused measurement image blocks; and reconstructing each of the plurality of fused measurement image blocks using a perception matrix to obtain the plurality of fused image blocks, wherein the perception matrix is ​​obtained by multiplying the measurement matrix by a redundant dictionary matching the measurement matrix.

2. The method according to claim 1, characterized in that, The process involves performing image transformations in multiple directions on at least one image to be fused, resulting in a low-frequency image group and a high-frequency image group corresponding to the at least one image, including: The at least one image to be fused is subjected to a four-way non-downsampled contourlet transform to obtain the low-frequency image group and the high-frequency image group.

3. The method according to claim 1, characterized in that, The matching degree between pixel-based region energies is used to fuse the pixels of each of the multiple observation image block pairs to obtain multiple fused measurement image blocks, including: Perform the following operations on each of the observed image block pairs as the current observed image block pair to obtain the plurality of fused measurement image blocks: Perform the following fusion operation on each pixel in the currently observed image patch pair as the current pixel: Determine the matching degree between the region energy of the current observed image patch and the current pixel point to obtain the current matching degree; If the current matching degree is greater than or equal to the matching degree threshold, the pixel values ​​of the current pixel point in the current observation image block pair are weighted and summed to obtain the pixel value of the current pixel point in the fused measurement image block corresponding to the current observation image block pair; If the current matching degree is less than the matching degree threshold, the pixel value of the target observation image block at the current pixel point is determined as the pixel value of the current pixel point in the fused measurement image block corresponding to the current observation image block pair, wherein the target observation image block is the observation image block with the largest regional energy at the current pixel point in the current observation image block pair.

4. The method according to claim 3, characterized in that, Before performing a weighted summation of the pixel values ​​of the current pixel point on the currently observed image patch, the method further includes: The product of the ratio of the first difference to the second difference and a preset coefficient is determined as the first weight, and the difference between 1 and the first weight is determined as the second weight. The first difference is the difference between the current matching degree and the matching degree threshold, and the second difference is the difference between 1 and the matching degree threshold. The first weight is the weighting coefficient corresponding to the observation image block with the largest pixel value of the current pixel in the current observation image block pair, and the second weight is the weighting coefficient corresponding to the observation image block with the smallest pixel value of the current pixel in the current observation image block pair.

5. The method according to claim 1, characterized in that, The process of reconstructing each of the plurality of fused measurement image blocks using a perception matrix to obtain the plurality of fused image blocks includes: The following reconstruction operation is performed on each column of each fused measurement image block using the perception matrix to obtain the plurality of fused image blocks, wherein each column is the current column during the following reconstruction operation: Construct sparse coefficients that match the current column using the perception matrix; The sparse coefficients are multiplied by the redundant dictionary to obtain the current column in the fused image block corresponding to each fused measurement image block.

6. The method according to any one of claims 1 to 5, characterized in that, The step of fusing images from the low-frequency image group to obtain a low-frequency fused image includes: The following fusion operation is performed on each pixel of the image in the low-frequency image group as the current pixel to obtain the pixel value of the current pixel in the low-frequency fused image: Determine the variance of pixel values ​​of a set of pixels in a preset neighborhood centered on the current pixel in each image of the low-frequency image group to obtain the neighborhood variance corresponding to each image. The largest neighborhood variance among the neighborhood variances corresponding to each of the images is determined as the pixel value of the current pixel in the low-frequency fused image.

7. An image fusion apparatus, characterized in that, include: The transformation unit is used to perform image transformations in multiple directions on at least one image to be fused, to obtain a low-frequency image group and a high-frequency image group corresponding to the at least one image, wherein the low-frequency image group includes low-frequency images corresponding to each image, and the high-frequency image group includes high-frequency images corresponding to each image. The processing unit is used to divide the images in the high-frequency image group into blocks according to a preset size to obtain multiple image block pairs corresponding to the high-frequency image group. The execution unit is configured to perform image block fusion on the image blocks in each of the plurality of image block pairs to obtain a plurality of fused image blocks, and to stitch the plurality of fused image blocks together to obtain a high-frequency fused image; A fusion unit is used to perform image fusion on the images in the low-frequency image group to obtain a low-frequency fused image; An inverse transform unit is used to perform an inverse image transform on the high-frequency fused image and the low-frequency fused image to obtain a fused image of the at least one image; The execution unit includes: a first execution module, configured to multiply the measurement matrix by each image block in each image block pair to obtain multiple observation image block pairs, wherein the number of columns in the measurement matrix is ​​the same as the number of rows in each image block pair; a fusion module, configured to fuse the pixels of each observation image block pair based on the matching degree between the regional energies of the pixels to obtain multiple fused measurement image blocks; and a reconstruction module, configured to reconstruct each fused measurement image block in the multiple fused measurement image blocks using a perception matrix to obtain the multiple fused image blocks, wherein the perception matrix is ​​obtained by multiplying the measurement matrix by a redundant dictionary matching the measurement matrix.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 6.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 6 through the computer program.

Images