Image fusion method and device, equipment and storage medium

Abstract

The invention discloses an image fusion method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the steps that a first image and a second image are acquired, the size of the first image is larger than that of the second image, and the exposure of the first image is smaller than that of the second image; and fusing the first image and the second image to obtain a fused image. According to the method, a first image with relatively small exposure and relatively large size is fused with a second image with relatively large exposure and relatively small size. While the high dynamic range image is obtained, the data transmission amount in the image fusion process is reduced, and the equipment power consumption is reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image fusion method, device, equipment and storage medium. Background Art

[0002] High Dynamic Range Video (HDR Video) refers to a video sequence composed of images with a wider color range, higher contrast and brightness. High dynamic range video technology can better restore the human eye's perception of high dynamic scenes in a dynamic form.

[0003] Image fusion technology can be used to fuse multiple images of the same scene with different exposures to obtain a high dynamic range image. In the terminal device, after the image sensor captures the image, it transfers it to the memory. When image fusion is required, the processor accesses the memory and calls these images, or these images are uploaded to the server.

[0004] In the related art, large-size images of the same size are often used for fusion, which results in a large amount of data transmission and high power consumption of the device in the above process. Summary of the invention

[0005] The embodiment of the present application provides an image fusion method, device, equipment and storage medium. The technical solution provided by the embodiment of the present application is as follows:

[0006] According to one aspect of an embodiment of the present application, an image fusion method is provided, the method comprising:

[0007] Acquire a first image and a second image, wherein a size of the first image is larger than a size of the second image, and an exposure amount of the first image is smaller than an exposure amount of the second image;

[0008] The first image and the second image are fused to obtain a fused image.

[0009] According to one aspect of an embodiment of the present application, an image fusion device is provided, the device comprising:

[0010] An acquisition module, used for acquiring a first image and a second image, wherein a size of the first image is larger than a size of the second image, and an exposure amount of the first image is smaller than an exposure amount of the second image;

[0011] The fusion module is used to fuse the first image and the second image to obtain a fused image.

[0012] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned image fusion method.

[0013] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned image fusion method.

[0014] According to one aspect of an embodiment of the present application, a chip is provided, wherein the chip includes a programmable logic circuit and / or program instructions, and when the chip is running, it is used to implement the above-mentioned image fusion method.

[0015] According to one aspect of an embodiment of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium, and a processor reading and executing the computer program from the computer-readable storage medium to implement the above-mentioned image fusion method.

[0016] The technical solution provided by the embodiments of the present application includes at least the following beneficial effects:

[0017] By fusing a first image with a smaller exposure and a larger size with a second image with a larger exposure and a smaller size, a high dynamic range image is obtained, while the amount of data transmission is reduced and the power consumption of the device is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0019] Figure 2 is a flow chart of an image fusion method provided by an embodiment of the present application;

[0020] Figure 3 is a flow chart of an image fusion method provided by another embodiment of the present application;

[0021] Figure 4 is a schematic diagram of image fusion through Laplacian pyramid decomposition provided by an embodiment of the present application;

[0022] Figure 5 is a block diagram of an image fusion device provided by an embodiment of the present application;

[0023] Figure 6 It is a structural block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0025] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application, and the solution implementation environment may include: a terminal device 10 and a server 20.

[0026] The terminal device 10 may be an electronic device such as a mobile phone, a tablet computer, a PC (Personal Computer), a vehicle-mounted terminal, a smart home appliance, a multimedia player, etc. The terminal device 10 may be installed with a client that runs a target application, and the target application may be an image processing application, a video recording application, etc., which is not limited in this application.

[0027] The server 20 can be used to provide background services for the client of the target application (such as the image processing application) in the terminal device 10. For example, the server 20 can be the background server of the above-mentioned target application (such as the image processing application). The server 20 can be a single server, a server cluster composed of multiple servers, or a cloud computing service center.

[0028] The terminal device 10 and the server 20 may communicate via a network, such as a wireless or wired network.

[0029] In the image fusion method provided in the embodiment of the present application, the execution subject of each step may be an electronic device. For example, the electronic device may be Figure 1 The terminal device 10 in the implementation environment of the solution shown may also be Figure 1 The server 20 in the implementation environment of the solution shown. In addition, for the convenience of description, the electronic device is used as the execution subject of each step in the following embodiments.

[0030] Please refer to Figure 2 , which shows a flow chart of an image fusion method provided by an embodiment of the present application. The method may include at least one of the following steps 210 to 220.

[0031] Step 210: Acquire a first image and a second image, wherein the size of the first image is larger than the size of the second image, and the exposure amount of the first image is smaller than the exposure amount of the second image.

[0032] The first image and the second image are two images of the same scene with different exposures and sizes. The exposure of an image is used to measure the amount of light that hits the image sensor during the image capture process. The size of an image refers to the product of the width and height of the image. In an embodiment of the present application, the width and height of the image are both in pixels. For example, if the width value of an image is w and the height value is h, the size of the image can be expressed as w×h, that is, the number of pixels in each row of the image is w, and the number of pixels in each column is h.

[0033] In some embodiments, the first image and the second image can be obtained by the terminal device by continuously shooting the same scene in a short time. For example, after the first shooting, the second shooting is performed at an interval of 0.5 seconds, and different exposures can be set for them when shooting. In other embodiments, the first image and the second image can also come from multiple frames of video with a short time interval in the video. For example, when recording a video, a frame of video frame image can be collected every 0.5 seconds.

[0034] During the image or video shooting process, the object being photographed will first generate an optical image through the lens (Lens), and then the optical image will be projected onto the surface of the image sensor and converted into an electrical signal. After that, after a series of analog-to-digital conversions and processing, the image that we can see is obtained. In some embodiments, when the image sensor captures multiple images of the same size for the same scene, these images can be binned to obtain a smaller image and retain an image of the original size. In the end, a larger image with a smaller exposure and several smaller images with a larger exposure are obtained, and the larger image can be selected as the first image, and one of the smaller images can be selected as the second image.

[0035] Exemplarily, one pixel merging method is to first add the induced charges of the pixel units in adjacent rows in the image captured by the image sensor, that is, to merge every two pixel units in adjacent rows into one pixel unit. Then, add the induced charges of the pixel units in adjacent columns, that is, to merge every two pixel units in adjacent columns into one pixel unit. Assuming that the size of the above image is w×h, after the above operation, an image of size w / 2×h / 2 can be obtained.

[0036] The size of the first image is larger than the size of the second image, which means that at least one of the width value and the height value of the first image is larger than the second image, and the other is larger than or equal to the second image. For example, if the width value of the first image is larger than the second image, and the height value of the first image is equal to the second image, then the size of the first image is larger than the second image. For another example, if the height value of the first image is larger than the second image, and the width value of the first image is equal to the second image, then the size of the first image is larger than the second image. For another example, if both the width value and the height value of the first image are larger than the second image, then the size of the first image is larger than the second image.

[0037] In some embodiments, the first image is downsampled a limited number of times and its size can be adjusted to the size of the second image. For example, if the width and height of the image are each reduced by half after downsampling, and the size of the first image is w×h, then the size of the second image can be w / 2×h / 2, or w / 4×h / 4, w / 8×h / 8, etc., which is not limited in the present application.

[0038] Step 220: Fuse the first image and the second image to obtain a fused image.

[0039] The fused image is an image with a higher dynamic range obtained by fusing the first image and the second image.

[0040] When shooting or photographing, if high-brightness areas under strong light sources and relatively low-brightness areas such as shadows and backlights exist in the image at the same time, the output image will have the problem that the bright areas become white due to overexposure, and the dark areas become black due to underexposure, resulting in missing image information.

[0041] High dynamic range images can retain the image information of the above-mentioned high-brightness areas and low-brightness areas as much as possible, and the brightness and color of the image are also more vivid.

[0042] The technical solution provided by the present application combines a first image with a smaller exposure and a larger size with a second image with a larger exposure and a smaller size, thereby obtaining a high dynamic range image while reducing the amount of data transmission and the power consumption of the device.

[0043] Please refer to Figure 3 In some embodiments, step 220 may include at least one of sub-steps 222 to 224 .

[0044] Step 222 , performing Laplacian pyramid decomposition on the first image and the second image respectively, to obtain a Gaussian pyramid and a Laplacian pyramid corresponding to the first image, and a Gaussian pyramid and a Laplacian pyramid corresponding to the second image.

[0045] The Gaussian pyramid is an image pyramid including multiple Gaussian images, and the Laplacian pyramid is an image pyramid including multiple Laplacian images. In the image pyramid, images are arranged in order of size from large to small or from small to large.

[0046] In some embodiments, when performing Laplacian pyramid decomposition on the first image, the first image may be decomposed based on the formula Construct a Gaussian pyramid; where G i is the Gaussian image of the i-th level, G i-1 is the Gaussian image of the i-1th level, the initial value of i is 1, G0 can correspond to the first image, DOWN represents the downsampling operation, represents convolution, K i is the filter coefficient of the filter. This formula indicates that the Gaussian image of the i-1th level can be obtained by first performing convolution filtering on the Gaussian image of the i-1th level and then performing downsampling. For the Gaussian image of the i-th level, a corresponding intermediate image can be obtained by first upsampling the Gaussian image of the i+1th level and then performing convolution filtering, that is, Among them, G′ i is the intermediate image corresponding to the Gaussian image of the i-th level, G i+1 is the Gaussian image of the i+1th level, UP represents the upsampling operation, K′ i is the filter coefficient, which can be i The same or different. Subtracting the corresponding intermediate image from the Gaussian image of the i-th level can obtain the Laplacian image of the i-th level, that is, the Laplacian image L of the i-th level i =G i -G′ i .

[0047] Please refer to Figure 4 In some embodiments, the process of performing n-layer Laplacian pyramid decomposition on the first image P0 can be regarded as first performing n convolution filtering and downsampling operations on the first image P0 to obtain n Gaussian images (G1, G2, ..., G n-1 , G n ), and then perform n upsampling and convolution filtering operations on these n Gaussian images to obtain n intermediate images ( Figure 4 Afterwards, the first image P0 and n-1 Gaussian images (G1, G2, ..., G n-1 ) are subtracted from the above n intermediate images at the same scale to obtain n Laplace images (L0, L1, L2, ..., L n-1 ). Finally, the n Gaussian images constitute a Gaussian pyramid corresponding to the first image, and the n Laplacian images constitute a Laplacian pyramid corresponding to the first image.

[0048] In addition, in the process of obtaining n Gaussian images in the above embodiment, when performing convolution filtering, a filter with the same filter coefficient can be used, or starting from obtaining the first Gaussian image G1, filters with lower filter coefficients than the previous level can be used in sequence. In the process of obtaining n intermediate images, when performing convolution filtering, the filter coefficient of the filter can be the same as the filter coefficient of the filter used in the process of obtaining n Gaussian images, or it can be different. For the filter coefficient of the filter in this application, the technician can freely set it based on the needs of image processing, and this application does not limit this.

[0049] Subtracting at the same scale means that the two images being subtracted have the same size. Therefore, in the above scheme, downsampling an image first and then upsampling it will not change the size of the image. For example, the downsampling operation on the image can be to delete the pixels in the even-numbered rows and columns in the image, thereby reducing the width and height of the image to half of the original size; the upsampling operation on the image can be to fill the pixels in the deleted even-numbered rows and columns in the image with 0, thereby restoring the image to its original size. The above-mentioned upsampling and downsampling methods are only used as examples in this application and are not limited to them.

[0050] The process of performing Laplacian pyramid decomposition on the second image can refer to the first image, but it should be noted that since the size of the second image is smaller than that of the first image, the number of layers of Laplacian pyramid decomposition on the second image must be smaller than that of the first image. For example, please refer to Figure 4 In some embodiments, the first image P0 is decomposed by an n-layer Laplacian pyramid, and the second image P1 is decomposed by an n-1-layer Laplacian pyramid. Then, the Gaussian pyramid corresponding to the second image includes n-1 Gaussian images (H2, ..., H n-1 , H n ), the Laplacian pyramid corresponding to the second image contains n-1 Laplacian images (M1, M2, ..., M n-1 ). In some other embodiments, if the first image is subjected to n-layer Laplacian pyramid decomposition, the second image may also be subjected to n-2, n-3, ... layers of Laplacian pyramid decomposition, which is not limited in the present application.

[0051] Step 224 , performing image fusion at the same scale according to the Gaussian pyramid and the Laplacian pyramid corresponding to the first image, and the Gaussian pyramid and the Laplacian pyramid corresponding to the second image, to obtain a fused image.

[0052] Performing image fusion at the same scale means that the sizes of the images involved in the fusion are the same during the image fusion process. In addition, in the embodiment of the present application, performing image fusion does not change the size of the image. For example, when two images of size w×h are fused, the size of the fused image is still w×h.

[0053] In some embodiments, step 224 may include the following sub-steps:

[0054] 1. Fuse the two Gaussian images with the smallest scale in the above two Gaussian pyramids to obtain the i-th order fused image, where the initial value of i is 1, and the two Gaussian images have the same scale;

[0055] The two Gaussian pyramids are respectively a Gaussian pyramid corresponding to the first image and a Gaussian pyramid corresponding to the second image. The two Gaussian images having the same scale means that the two Gaussian images have the same size.

[0056] For example, please refer to Figure 4 In some embodiments, the Gaussian image with the smallest scale in the Gaussian pyramid corresponding to the first image P0 is G n , the Gaussian image with the smallest scale in the Gaussian pyramid corresponding to the second image P1 is H n , G n and H n With the same scale, they are fused to obtain the i-th order fused image Qn.

[0057] In the above example, since the second image P1 is decomposed by n-1 layers of Laplacian pyramid, the corresponding Gaussian pyramid can be found n Gaussian images H with the same scale n In fact, the two Gaussian images with the smallest scales in the Gaussian pyramid corresponding to the first image and the Gaussian pyramid corresponding to the second image may not have the same scale. For example, please refer to Figure 4 , if only the second image P1 is decomposed by n-2 layers of Laplacian pyramid, then the Gaussian image with the smallest scale in its corresponding Gaussian pyramid is composed of H n Changed to H n-1 At this time, the Gaussian image G n-1 With Gaussian image H n-1 The fusion is performed to obtain the i-th order fused image.

[0058] In some embodiments, the two Gaussian images may be fused by the following steps:

[0059] (1) Perform brightness alignment on two Gaussian images to obtain two aligned Gaussian images.

[0060] In the two Gaussian images, the Gaussian image decomposed from the first image and the Gaussian image decomposed from the second image have a huge brightness difference due to the different exposure. In order to facilitate image fusion, it is necessary to adjust the brightness of the two images so that their brightness is generally similar. This process is called brightness alignment.

[0061] In some embodiments, the brightness alignment can be performed by multiplying the brightness ratio of the Gaussian image obtained by decomposing the first image to the Gaussian image obtained by decomposing the second image, that is, G N_luma =G N ·exp ratio , where G N is the Gaussian image decomposed from the first image in the two images, exp ratio is the brightness ratio mentioned above, G N_luma G N The image obtained after brightness alignment, this formula can be understood as N Each pixel in the image is multiplied by the same brightness ratio, so the overall brightness of the two aligned Gaussian images is similar, but there are still differences in the brightness of some areas. In the above scheme, the brightness of the Gaussian image obtained by decomposing the second image does not change after brightness alignment. In other embodiments, the brightness of both Gaussian images can also be adjusted at the same time, which is not limited in this application.

[0062] (2) According to the motion area, overexposed area and underexposed area in the two aligned Gaussian images, the fusion weights corresponding to the two aligned Gaussian images are determined.

[0063] The above-mentioned motion area refers to the area where there is image change in the two Gaussian images. The overexposed area refers to the area in the image where the highlight details are missing due to overexposure, and the underexposed area refers to the area in the image where the dark details are missing due to lack of exposure. The fusion weight corresponding to the image refers to the proportion of the image in the fused image.

[0064] In some embodiments, the fusion weight w corresponding to the image with smaller exposure in the two aligned Gaussian images can be determined by the following formula: GN : Among them, k1, k2 and k3 are hyperparameters obtained through empirical debugging. H The image H with larger exposure in the two aligned Gaussian images N Normalized brightness, R is the brightness ratio, RG N is the image with smaller exposure in the two aligned Gaussian images. In the above formula, Represents the judgment of overexposed and underexposed areas. Indicates the judgment of the motion area. In addition, in the two aligned Gaussian images, the fusion weight corresponding to the image with larger exposure is 1-w GN .

[0065] It should be noted that, in the embodiment of the present application, the exposure of the Gaussian image decomposed from the first image is always less than the exposure of the Gaussian image decomposed from the second image, and the exposure of the Laplacian image decomposed from the first image is always less than the exposure of the Laplacian image decomposed from the second image. Moreover, after brightness alignment, the above size relationship will not change.

[0066] In addition, it should be noted that in the embodiments of the present application, the formulas involved in calculating the fusion weights are only exemplary, and technicians can change the formulas according to needs, such as adding parameter bias or multiplication coefficients, replacing the exponential function with an inverse function, etc. For example, the formula for the fusion weight corresponding to the image with the smaller exposure in the above-mentioned two aligned Gaussian images can also be:

[0067]

[0068] Among them, k1, k2, k3 and k4 are all hyperparameters, which are obtained based on experience.

[0069] (3) According to the fusion weights corresponding to the two aligned Gaussian images, the two aligned Gaussian images are fused to obtain an i-th order fused image.

[0070] If you use Q N represents the i-th order fused image, then the above fusion method is Q N =w GN ·RG N +(1-w GN )·H N .

[0071] By adopting the above method, a fusion weight can be determined for each pixel in the two aligned Gaussian images. When performing image fusion, as many images with smaller exposure as possible can be selected in overexposed areas and motion areas, and as many images with larger exposure as possible can be selected in underexposed areas, so that the fused image retains more image information and improves the dynamic range of the fused image.

[0072] 2. Upsampling and filtering the i-th order fused image to obtain a processed i-th order fused image; wherein the processed i-th order fused image has the same scale as the Laplacian images with the smallest scale in the two Laplacian pyramids.

[0073] The processed i-th order fused image is used to fuse with the Laplacian images with the same scale in two Laplacian pyramids.

[0074] 3. The processed i-th order fused image is fused with two Laplacian images having the same scale as the processed i-th order fused image in two Laplacian pyramids to obtain an i+1-th order fused image.

[0075] The two Laplacian images having the same scale as the processed i-th order fused image refer to images having the same size as the i-th order fused image after upsampling and filtering in the Laplacian pyramid corresponding to the first image and the Laplacian pyramid corresponding to the second image.

[0076] For example, please refer to Figure 4 , if the i-th order fused image is Q n , then after upsampling and filtering, it is combined with L n-1 and M n-1 Fusion is performed; if the i-th order fused image is Q2, it is upsampled and filtered and then fused with L1 and M1; if the i-th order fused image is Q1, since after upsampling and filtering, there is no Laplacian image with the same scale in the Laplacian pyramid corresponding to the second image, it is upsampled and filtered and then only fused with L0.

[0077] In some embodiments, the i+1th order fused image may be obtained by fusion through the following steps:

[0078] (1) Brightness alignment is performed on two Laplacian images in two Laplacian pyramids that have the same scale as the processed i-th order fusion image to obtain two aligned Laplacian images.

[0079] The specific method of performing brightness alignment on the Laplacian image can refer to the method of performing brightness alignment on the Gaussian image in the above embodiment, which will not be described in detail here.

[0080] (2) According to the fusion weights corresponding to the two aligned Gaussian images, the fusion weights corresponding to the two aligned Laplacian images are determined.

[0081] In some embodiments, this step can be implemented by:

[0082] First, the fusion weight corresponding to the first Gaussian image of the two aligned Gaussian images is upsampled to obtain a first weight parameter.

[0083] The first Gaussian image is any one of the two aligned Gaussian images, and the first weight parameter is used to participate in the weighted sum to determine the fusion weight corresponding to the first Laplacian image.

[0084] In some embodiments, the first Gaussian image is the image with the smaller exposure among the two aligned Gaussian images, and the first weight parameter w1=UP(w GN ).

[0085] Since there is a fusion weight for each pixel in the image in the embodiment of the present application, the fusion weight corresponding to the image is actually a fusion weight matrix. In some embodiments, the upsampling process of the fusion weight can add additional even rows and columns to the fusion weight matrix, and fill the even rows and columns with odd rows and columns of fusion weights, and finally obtain the first weight parameter matrix. For the method of upsampling the fusion weight, this application is only used as an exemplary description and is not limited to this.

[0086] Secondly, the second weight parameter is determined according to the brightness and signal-to-noise ratio respectively corresponding to the two aligned Laplacian images.

[0087] The second weight parameter is used to participate in the weighted sum to determine the fusion weight corresponding to the first Laplacian image.

[0088] In some embodiments, the signal-to-noise ratios corresponding to the two Laplacian images can be determined by using a pre-calibrated noise model according to the brightness corresponding to the two aligned Laplacian images. For example, a noise model can be established by shooting a 24 or 140 color card and counting the noise variance of color blocks of different brightnesses. Through this model, the signal-to-noise ratios corresponding to different brightnesses can be queried.

[0089] In some embodiments, the second weight parameter may be determined according to the signal-to-noise ratios of the two aligned Laplacian images and the brightness of the Laplacian image with a larger exposure in the two aligned Laplacian images. For example, the second weight parameter may be determined by the following formula: w2 = max(1 - Among them, w2 is the second weight parameter, S1 is the signal-to-noise ratio of the Laplace image with larger exposure in the two aligned Laplace images, S2 is the signal-to-noise ratio of the Laplace image with smaller exposure in the two aligned Laplace images, h is the hyperparameter, Luma H ′ is the normalized brightness corresponding to the Laplacian image with larger exposure in the two aligned Laplacian images. In the above formula, It represents the judgment of the level of image noise. When the difference between S1 and S1 is too large, it can be regarded as a high noise area. Represents the judgment of the brightness of the image.

[0090] Then, a weighted sum is performed on the first weight parameter and the second weight parameter to obtain a fusion weight corresponding to the first Laplacian image in the two aligned Laplacian images.

[0091] If the first Gaussian image is the image with the smaller exposure among the two aligned Gaussian images, then the first Laplace image is the image with the smaller exposure among the two aligned Laplace images; if the first Gaussian image is the image with the larger exposure among the two aligned Gaussian images, then the first Laplace image is the image with the larger exposure among the two aligned Laplace images. For example, the formula w G =r·w1+(1-r)·w2 to obtain the fusion weight corresponding to the first Laplacian image. G is the fusion weight corresponding to the first Laplace image, and r is the variable weight.

[0092] Finally, based on the fusion weight corresponding to the first Laplace image, the fusion weight corresponding to the second Laplace image in the two aligned Laplace images is obtained.

[0093] If the first Laplace image is the image with the smaller exposure among the two aligned Laplace images, the second Laplace image is the image with the larger exposure among the two aligned Laplace images; if the first Laplace image is the image with the larger exposure among the two aligned Laplace images, the second Laplace image is the image with the smaller exposure among the two aligned Laplace images.

[0094] For example, if the fusion weight corresponding to the first Laplacian image is w G , then the fusion weight corresponding to the second Laplacian image is 1-w G .

[0095] Generally, the human eye is more sensitive to noise in dark areas of an image than in bright areas. Therefore, images with larger exposure and higher signal-to-noise ratio need to be selected in low-brightness areas of the image. The above method introduces the second weight parameter, so that when performing image fusion, more images with larger exposure can be selected in high-noise or low-brightness areas, avoiding the introduction of too much noise from the first image during the image fusion process, thereby improving the signal-to-noise ratio of the final output fused image. In addition, by introducing the first weight parameter, when performing image fusion, more images with smaller exposure and higher resolution can be selected in high-brightness areas, taking into account the requirements for resolution in high-brightness areas, and ensuring the resolution and clarity of the fused image.

[0096] (3) According to the fusion weights corresponding to the two aligned Laplacian images, the processed i-th order fused image and the two aligned Laplacian images are fused to obtain an i+1-th order fused image.

[0097] For example, please refer to Figure 4, if the first Laplace image is the image with the smaller exposure among the two aligned Laplace images, the second Laplace image is the image with the larger exposure among the two aligned Laplace images, and the i-th order fused image is Q n , then the i+1th order fused image can be expressed as Q n-1 , the first Laplacian image can be expressed as L n-1 , the second Laplace image can be expressed as M n-1 And, through the formula Q n-1 =UP(Q n )+w G L n-1 +(1-w G )M n-1 The i+1th order fusion image can be obtained, where UP(Q n ) represents the Q after upsampling and filtering n .

[0098] By using the above method, multiple images are fused to obtain an i+1th order fused image, which can make the i+1th order fused image have as much image information as possible, thereby improving the dynamic range of the fused image finally outputted.

[0099] 4. If there is a Laplacian image with a scale larger than the i+1th order fused image in at least one Laplacian pyramid, let i=i+1, and start again from the step of upsampling and filtering the i-th order fused image to obtain the processed i-th order fused image.

[0100] If there is no Laplacian image with a scale greater than the i+1th order fused image in the two Laplacian pyramids, the i+1th order fused image is determined as the fused image.

[0101] For example, please refer to Figure 4 , and then get the i+1th order fusion image Q n-1 After that, the Laplacian pyramid corresponding to the first image P0 and the Laplacian pyramid corresponding to the second image P1 both have a scale greater than Q n-1 Laplace image of Q n-1 As the i-th order image, the process starts from the step of upsampling and filtering the i-th order fused image to obtain the processed i-th order fused image. By looping the above operations, when Q1 is taken as the i-th order fused image and the i+1-th order fused image Q0 is obtained, in particular, since there is no Laplacian image with a scale larger than Q1 in the Laplacian pyramid corresponding to the second image P1, only the fusion weight corresponding to the Laplacian image L0 with a smaller exposure amount needs to be considered. If w is used G0 represents the fusion weight, then Q0=UP(Q1)+w G0L0, where UP(Q1) represents Q1 after upsampling and filtering, and w G0 It can be directly obtained by sampling the fusion weight of the Laplacian image L1 of the next level in the previous cycle. After obtaining Q0 as the i+1th order fusion image, since there is no Laplacian image with a scale greater than Q0 in the Laplacian pyramid corresponding to the first image P0 and the Laplacian pyramid corresponding to the second image P1, Q0 is determined as the fused image.

[0102] The above method performs multi-scale Laplacian pyramid decomposition on the first image and the second image, and only performs image fusion at the same scale, which can save device power consumption and avoid information loss caused by fusion of images at different scales.

[0103] The image fusion method provided in the embodiment of the present application can also be applied to the scene of fusing more than two images of different sizes. For example, if there are three images with different exposures for the same scene, namely, image a, image b and image c, and the size of image a is larger than that of image b, the size of image b is larger than that of image c, the exposure of image a is smaller than that of image b, and the exposure of image b is smaller than that of image c. Then, image a can be used as the first image and image b as the second image to obtain a fused image, and then the fused image can be used as the first image and image c as the second image, and finally a fused image of image a, image b and image c can be obtained. In addition, it should be noted that the present application does not limit the specific order of multi-image fusion. For example, in the above scenario, image b can be used as the first image and image c can be used as the second image, and then image a can be used as the first image and the fused image of image b and image c can be used as the second image.

[0104] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0105] Please refer to Figure 5 , which shows a block diagram of an image fusion device provided by an embodiment of the present application. The device has the function of implementing the above-mentioned image fusion method, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be an electronic device, or can be set in an electronic device. The device 500 can include: an acquisition module 510 and a fusion module 520.

[0106] The acquisition module 510 is used to acquire a first image and a second image, wherein the size of the first image is larger than the size of the second image, and the exposure amount of the first image is smaller than the exposure amount of the second image.

[0107] The fusion module 520 is used to fuse the first image and the second image to obtain a fused image.

[0108] In some embodiments, the fusion module 520 includes: a decomposition submodule and a fusion submodule.

[0109] The decomposition submodule is used to perform Laplacian pyramid decomposition on the first image and the second image respectively to obtain a Gaussian pyramid and a Laplacian pyramid corresponding to the first image, and a Gaussian pyramid and a Laplacian pyramid corresponding to the second image.

[0110] The fusion submodule is used to perform image fusion on the same scale according to the Gaussian pyramid and the Laplacian pyramid corresponding to the first image and the Gaussian pyramid and the Laplacian pyramid corresponding to the second image to obtain the fused image.

[0111] In some embodiments, the fusion submodule includes: a first fusion unit, a processing unit, a second fusion unit and a determination unit.

[0112] The first fusion unit is used to fuse the two Gaussian images with the smallest scales in the two Gaussian pyramids to obtain an i-th order fused image, where the initial value of i is 1, and the two Gaussian images have the same scale.

[0113] The processing unit is used to perform upsampling and filtering processing on the i-th order fused image to obtain a processed i-th order fused image; wherein the processed i-th order fused image has the same scale as the Laplacian images with the smallest scale in the two Laplacian pyramids.

[0114] The second fusion unit is used to fuse the processed i-th order fused image with two Laplacian images in the two Laplacian pyramids that have the same scale as the processed i-th order fused image to obtain an (i+1)-th order fused image.

[0115] The processing unit is further configured to, if there is a Laplacian image with a scale greater than that of the i+1th order fused image in at least one of the Laplacian pyramids, set i=i+1, and again start from the step of upsampling and filtering the i-th order fused image to obtain a processed i-th order fused image.

[0116] The determining unit is configured to determine the (i+1)th order fused image as the fused image if there is no Laplacian image with a scale greater than the (i+1)th order fused image in the two Laplacian pyramids.

[0117] In some embodiments, the first fusion unit is used to: perform brightness alignment on the two Gaussian images to obtain two aligned Gaussian images; determine the fusion weights corresponding to the two aligned Gaussian images according to the motion areas, overexposed areas and underexposed areas in the two aligned Gaussian images; and fuse the two aligned Gaussian images according to the fusion weights corresponding to the two aligned Gaussian images to obtain the i-th order fused image.

[0118] In some embodiments, the second fusion unit includes: a brightness alignment subunit, a weight determination subunit and an image fusion subunit.

[0119] The brightness alignment subunit is used to perform brightness alignment on two Laplacian images in the two Laplacian pyramids that have the same scale as the processed i-th order fused image to obtain two aligned Laplacian images.

[0120] The weight determination subunit is used to determine the fusion weights corresponding to the two aligned Laplacian images according to the fusion weights corresponding to the two aligned Gaussian images.

[0121] The image fusion subunit is used to fuse the processed i-th order fused image and the two aligned Laplacian images according to the fusion weights respectively corresponding to the two aligned Laplacian images to obtain the i+1-th order fused image.

[0122] In some embodiments, the weight determination subunit is used to: upsample the fusion weight corresponding to the first Gaussian image in the two aligned Gaussian images to obtain a first weight parameter; determine the second weight parameter according to the brightness and signal-to-noise ratio respectively corresponding to the two aligned Laplace images; perform weighted summation of the first weight parameter and the second weight parameter to obtain the fusion weight corresponding to the first Laplace image in the two aligned Laplace images; and obtain the fusion weight corresponding to the second Laplace image in the two aligned Laplace images based on the fusion weight corresponding to the first Laplace image.

[0123] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0124] Please refer to Figure 6, which exemplarily shows a structural block diagram of an electronic device provided by an embodiment of the present application.

[0125] Typically, the electronic device 600 includes a processor 601 and a memory 602 .

[0126] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI processor, which is used to process computing operations related to machine learning.

[0127] The memory 602 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 stores a computer program, which is loaded and executed by the processor 601 to implement the above-mentioned image fusion method.

[0128] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the electronic device 600, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0129] In some embodiments, a computer-readable storage medium is further provided, wherein a computer program is stored in the storage medium, and the computer program is loaded and executed by a processor to implement the above-mentioned image fusion method.

[0130] Optionally, the computer readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives) or optical disks, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0131] In some embodiments, a computer program product is also provided, the computer program product comprising a computer program, the computer program is stored in a computer-readable storage medium, and a processor reads and executes the computer program from the computer-readable storage medium to implement the above-mentioned image fusion method.

[0132] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0133] The above are merely exemplary embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image fusion method, characterized in that: The method comprises: Acquire a first image and a second image, wherein a size of the first image is larger than a size of the second image, and an exposure amount of the first image is smaller than an exposure amount of the second image; The first image and the second image are fused to obtain a fused image.

2. The method according to claim 1, characterized in that: The fusing the first image and the second image to obtain a fused image includes: Performing Laplacian pyramid decomposition on the first image and the second image respectively to obtain a Gaussian pyramid and a Laplacian pyramid corresponding to the first image, and a Gaussian pyramid and a Laplacian pyramid corresponding to the second image; According to the Gaussian pyramid and the Laplacian pyramid corresponding to the first image, and the Gaussian pyramid and the Laplacian pyramid corresponding to the second image, image fusion is performed on the same scale to obtain the fused image.

3. The method according to claim 2, characterized in that The step of fusing the images on the same scale according to the Gaussian pyramid and the Laplacian pyramid corresponding to the first image and the Gaussian pyramid and the Laplacian pyramid corresponding to the second image to obtain the fused image includes: Fusing the two Gaussian images with the smallest scales in the two Gaussian pyramids to obtain an i-th order fused image, where the initial value of i is 1, and the two Gaussian images have the same scale; Performing upsampling and filtering processing on the i-th order fused image to obtain a processed i-th order fused image; wherein the processed i-th order fused image has the same scale as the Laplacian images with the smallest scale in the two Laplacian pyramids; fusing the processed i-th order fused image with two Laplacian images in the two Laplacian pyramids that have the same scale as the processed i-th order fused image to obtain an (i+1)th order fused image; If there is a Laplacian image with a scale larger than the i+1th order fused image in at least one of the Laplacian pyramids, let i=i+1, and start again from the step of upsampling and filtering the i-th order fused image to obtain a processed i-th order fused image; If there is no Laplacian image with a scale greater than that of the (i+1)th order fused image in the two Laplacian pyramids, the (i+1)th order fused image is determined as the fused image.

4. The method according to claim 3, characterized in that The step of fusing the two Gaussian images with the smallest scales in the two Gaussian pyramids to obtain an i-th order fused image includes: Performing brightness alignment on the two Gaussian images to obtain two aligned Gaussian images; Determining fusion weights corresponding to the two aligned Gaussian images according to the motion area, the overexposed area, and the underexposed area in the two aligned Gaussian images; The two aligned Gaussian images are fused according to the fusion weights respectively corresponding to the two aligned Gaussian images to obtain the i-th order fused image.

5. The method according to claim 4, characterized in that The step of fusing the processed i-th order fused image with two Laplacian images in the two Laplacian pyramids having the same scale as the processed i-th order fused image to obtain an i+1-th order fused image comprises: Performing brightness alignment on two Laplacian images in the two Laplacian pyramids that have the same scale as the processed i-th order fused image to obtain two aligned Laplacian images; Determining the fusion weights corresponding to the two aligned Laplacian images according to the fusion weights corresponding to the two aligned Gaussian images; The processed i-th order fused image and the two aligned Laplacian images are fused according to the fusion weights respectively corresponding to the two aligned Laplacian images to obtain the i+1-th order fused image.

6. The method according to claim 5, characterized in that The determining, according to the fusion weights respectively corresponding to the two aligned Gaussian images, the fusion weights respectively corresponding to the two aligned Laplacian images comprises: Upsampling the fusion weight corresponding to the first Gaussian image of the two aligned Gaussian images to obtain a first weight parameter; Determining a second weight parameter according to the brightness and signal-to-noise ratio respectively corresponding to the two aligned Laplacian images; Performing a weighted summation on the first weight parameter and the second weight parameter to obtain a fusion weight corresponding to the first Laplacian image in the two aligned Laplacian images; Based on the fusion weight corresponding to the first Laplacian image, a fusion weight corresponding to the second Laplacian image in the two aligned Laplacian images is obtained.

7. An image fusion device, characterized in that: The device comprises: An acquisition module, used for acquiring a first image and a second image, wherein a size of the first image is larger than a size of the second image, and an exposure amount of the first image is smaller than an exposure amount of the second image; The fusion module is used to fuse the first image and the second image to obtain a fused image.

8. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method according to any one of claims 1 to 6.

10. A chip, characterized in that: The chip comprises a programmable logic circuit and / or program instructions, and when the chip is running, it is used to implement the method according to any one of claims 1 to 6.

Images