Image fusion method, device, equipment and storage medium thereof
By acquiring and processing the histogram of HDR images and adjusting the fusion weight calculation formula, the problem of loss of details and unnatural transitions in highlights or dark areas in HDR images is solved, and the image quality is improved.
Patent Information
- Application Number
- CN202111525658.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-14
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-12-14
AI Technical Summary
During HDR image fusion, problems may lead to loss of details in highlights or dark areas and unnatural transitions, affecting image quality.
By obtaining the histogram of the original image, extracting histogram features to calculate parameters, adjusting the fusion weight calculation formula, and obtaining downsampling and upsampling fusion weights to ensure the consistency of image resolution, thereby generating high-quality fusion images.
The adaptability of fusion weight calculation is improved, and the problem of unsatisfactory brightness values in different scenarios is prevented. Highlights and dark details are maintained, noise is reduced, and the natural effect of image transition is achieved, which improves the quality of fusion images.
Smart Images

Figure CN114187213B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to an image fusion method and an apparatus, device and storage medium thereof. Background Art
[0002] HDR images refer to high-dynamic range images. At present, the fusion of HDR images usually designs a fixed fusion weight calculation formula, and the image information is brought into the fusion weight calculation formula point by point to calculate the fusion weight. Due to the differences in the performance of the same information in the image, such as brightness information, in different scenes and the influence of noise, the calculated fusion weight may deviate from the ideal value to a certain extent; the change trend of the weight obtained point by point is large, the brightness gradient in the transition interval may be too large, and the transition may be unnatural. In practical applications, the introduction of calculations in the HDR image fusion process may lead to the loss of highlights or dark details and unnatural transitions, which greatly affects the image quality.
[0003] The Chinese patent with publication number CN111462031 A discloses a multi-frame HDR image processing method, device, storage medium and electronic device. By acquiring at least two original images with different exposure levels, the acquired original images are pre-processed respectively to obtain weighted fusion images, and then the weighted fusion images are fused to obtain the final fusion image. The fusion weights are obtained by analyzing the detail of multiple frames of images, so as to fuse the multiple frames of images. However, the invention does not solve the problem that the introduction of calculations in the HDR image fusion process may cause the loss of highlights or dark details and unnatural transitions.
[0004] Therefore, it is necessary to provide an image fusion method and its device, equipment and storage medium to solve the above-mentioned problems in the prior art. Summary of the invention
[0005] The purpose of the present invention is to provide an image fusion method and its device, equipment and storage medium to solve the problem that the introduction of calculation in the HDR image fusion process may cause the loss of highlight or dark details and unnatural transition.
[0006] To achieve the above object, the image fusion method of the present invention comprises the steps of:
[0007] Providing an original image, and obtaining a histogram of the original image;
[0008] Extracting features of the histogram to obtain histogram calculation parameters, and adjusting and determining a fusion weight calculation formula according to the histogram calculation parameters;
[0009] Acquire brightness information in the original image, acquire a downsampled image according to the brightness information and the original image, and acquire a downsampled fusion weight according to the pixel brightness of the downsampled image and the fusion weight calculation formula;
[0010] Obtaining an upsampling fusion weight according to the downsampling fusion weight, so that a resolution of an image corresponding to the upsampling fusion weight is consistent with a resolution of the original image;
[0011] A fused image is obtained according to the up-sampling fusion weight and the original image.
[0012] The beneficial effects of the image fusion method of the present invention are:
[0013] By obtaining the histogram of the original image and the histogram calculation parameters, when image fusion is performed in different scenes, the fusion weight calculation formula is adjusted and determined through the histogram calculation parameters, and the statistical information of the histogram is fully utilized, so that the fusion weight calculation formula has higher adaptability, thereby improving the applicability of the image fusion method of the present invention, and effectively preventing the problem of unsatisfactory fusion weight calculation caused by different relative performances in different scenes or the same brightness value, thereby improving the problems of highlight overexposure, dark information loss, and noise abnormality caused by it; the upsampling fusion weight of the original resolution is upsampled according to the downsampling weight calculated by the downsampling image, which reflects the overall information of the image, and the fused image is obtained by the upsampling fusion weight and the original image, so as to obtain the local and overall information of the image, and the local information and the overall information of the image are combined, so as to achieve the effect of maintaining the image highlights and dark details, reducing noise, and natural image transition, thereby improving the quality of the fused image; and the problem that the introduction of calculation in the HDR image fusion process may cause the loss of highlights or dark details and unnatural transition is solved.
[0014] Optionally, the step of obtaining a histogram of the original image includes:
[0015] In several different scenes, the original images in each scene at different exposure times are obtained respectively, and the histogram of the original image is obtained. The beneficial effect is that the histogram of the original image at different exposure times in different scenes is obtained, so that the image fusion method is applicable to a variety of different scenes, thereby improving the applicability of the image fusion method of the present invention.
[0016] Optionally, the histogram calculation parameters include the median, integral and mean of the histogram.
[0017] Optionally, the fusion weight calculation formula includes a weight calculation curve, and the step of adjusting and determining the fusion weight calculation formula according to the histogram calculation parameters includes:
[0018] A preset weight curve, wherein a first slope of the weight curve in a first defined interval is less than or equal to 0 and is monotonically decreasing, a second slope of the weight curve in a second defined interval is less than or equal to 0 and is monotonically increasing, and a right endpoint of the first defined interval is connected to a left endpoint of the second defined interval;
[0019] Record the point where the first slope changes from 0 to non-0 as a first inflection point, and record the point where the second slope changes from non-0 to 0 as a second inflection point;
[0020] The shape of the weight curve, the first slope, the second slope, at least one of the coordinates of the first inflection point, and the coordinates of the second inflection point are adjusted according to the histogram calculation parameters to obtain the weight calculation curve. The beneficial effect is that the shape of the weight curve, the first slope, the second slope, at least one of the coordinates of the first inflection point, and the coordinates of the second inflection point are adjusted according to the histogram calculation parameters to obtain the weight calculation curve, so that the weight calculation curve is suitable for weight calculation of the original image corresponding to the histogram.
[0021] Optionally, the down-sampling fusion weight includes a first exposure image weight and a second exposure image weight, and the step of obtaining the down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula includes:
[0022] Traversing all pixels of the downsampled image to obtain brightness of the pixels, inputting the brightness of the pixels into the weight calculation curve to obtain a first weight result, normalizing the first weight result to obtain the first exposure image weight and the second exposure image weight, wherein the sum of the first exposure image weight and the second exposure image weight is 1;
[0023] The first exposure image weight corresponds to the first exposure image, the second exposure image weight corresponds to the second exposure image, and the exposure time of the first exposure image is greater than the exposure time of the second exposure image.
[0024] Optionally, the fusion weight calculation formula includes a one-dimensional weight calculation Gaussian function, and the step of adjusting and determining the fusion weight calculation formula according to the histogram calculation parameters includes:
[0025] A one-dimensional Gaussian function is preset, and the one-dimensional Gaussian function is adjusted according to the histogram calculation parameters to obtain the one-dimensional weight calculation Gaussian function. The beneficial effect is that by presetting the one-dimensional Gaussian function and adjusting the one-dimensional Gaussian function according to the histogram calculation parameters, the one-dimensional weight calculation Gaussian function obtained can be better applied to the weight calculation of the original image corresponding to the histogram.
[0026] Optionally, the step of obtaining the down-sampling fusion weight according to the pixel brightness of the down-sampled image and the fusion weight calculation formula includes:
[0027] The brightness of each pixel in the downsampled image is input into the one-dimensional weight calculation Gaussian function to obtain a second weight result, and the second weight result is normalized to obtain the downsampled fusion weight, wherein the downsampled fusion weight includes N weights, where N is a positive integer.
[0028] Optionally, the step of acquiring a downsampled image according to the brightness information and the original image includes:
[0029] The original image is divided into m×n blocks, where m and n are both positive integers, the average brightness of all pixels in each block is calculated, and the brightness of all pixels in each block is replaced by the average brightness corresponding to the block to obtain the downsampled image with a resolution of m×n.
[0030] Optionally, obtaining the upsampling fusion weight according to the downsampling fusion weight comprises the steps of:
[0031] The down-sampled fusion weight is up-sampled by bilinear interpolation to obtain the up-sampled fusion weight.
[0032] Optionally, obtaining the upsampling fusion weight according to the downsampling fusion weight comprises the steps of:
[0033] The down-sampled fusion weight is up-sampled by a two-dimensional Gaussian function to obtain the up-sampled fusion weight.
[0034] The present invention also provides an image fusion device, comprising:
[0035] An image processing module, used to provide an original image, obtain a histogram and brightness information of the original image, and obtain a downsampled image based on the brightness information and the original image;
[0036] An adjustment module, used for extracting the features of the histogram to obtain histogram calculation parameters, and adjusting and determining a fusion weight calculation formula according to the histogram calculation parameters;
[0037] A weight calculation module, used to obtain a down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula, and to obtain an up-sampling fusion weight according to the down-sampling fusion weight, so that the resolution of the image corresponding to the up-sampling fusion weight is consistent with the resolution of the original image;
[0038] The image fusion calculation module is used to obtain a fused image according to the up-sampling fusion weight and the original image.
[0039] The beneficial effects of the image fusion device of the present invention are:
[0040] The histogram and brightness information of the original image are obtained through the image processing module, and a downsampled image is obtained based on the brightness information and the original image, the histogram calculation parameters are obtained through the adjustment module, and the fusion weight calculation formula is adjusted and determined based on the histogram calculation parameters, the downsampled fusion weight and the upsampled fusion weight are obtained through the weight calculation module, and the image fusion calculation module obtains the fused image based on the upsampled fusion weight and the original image; the statistical information of the histogram is fully utilized, the applicability of the fusion weight calculation formula is improved, and the applicable scenarios of the image fusion method of the present invention are expanded; by calculating the downsampled image, the downsampled fusion weight and the upsampled fusion weight, the overall image and the local image are combined, so as to achieve the effects of maintaining the image highlights and dark details, reducing noise, and making the image transition natural, thereby improving the quality of the fused image; and the problem that the introduction of calculation in the HDR image fusion process may cause the loss of highlights or dark details and unnatural transition is solved.
[0041] The present invention also provides a device, comprising a memory, a processor and a program stored in the memory and executable on the processor, wherein the image fusion method is implemented when the processor executes the program.
[0042] The present invention also provides a storage medium on which a program is stored, and when the program is executed by a processor, the image fusion method is implemented.
[0043] The beneficial effects of the device and storage medium of the present invention are:
[0044] By adjusting and determining the fusion weight calculation formula through the histogram, the statistical information of the histogram is fully utilized, the applicability of the fusion weight calculation formula is improved, the applicable scenarios of the image fusion method of the present invention are expanded, and the overall and local images are combined by calculating the downsampled image, the downsampled fusion weight and the upsampled fusion weight, so as to achieve the effects of maintaining the image highlights and dark details, reducing noise, and natural image transition, thereby improving the quality of the fused image and solving the problem that the introduction of calculation in the HDR image fusion process may cause the loss of highlights or dark details and unnatural transition. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 is a flow chart of an image fusion method according to an embodiment of the present invention;
[0046] Figure 2 A schematic diagram of a weight curve according to an embodiment of the present invention;
[0047] Figure 3 is a schematic diagram of a bilinear interpolation method according to an embodiment of the present invention;
[0048] Figure 4 A schematic diagram of a grid of a bilinear interpolation method according to an embodiment of the present invention;
[0049] Figure 5 A weighted surface gradient graph formed by a two-dimensional Gaussian function according to an embodiment of the present invention;
[0050] Figure 6 FIG. 4 is a structural block diagram of an image fusion device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.
[0052] In view of the problems existing in the prior art, the embodiments of the present invention provide an image fusion method and its apparatus, device and storage medium to solve the problem that the introduction of calculation in the HDR image fusion process may cause the loss of highlight or dark details and unnatural transition.
[0053] Figure 1 The figure is a flow chart of the image fusion method according to an embodiment of the present invention.
[0054] In some embodiments of the present invention, referring to Figure 1 , the image fusion method comprises the steps of:
[0055] S0: providing an original image and obtaining a histogram of the original image;
[0056] S1: extracting the features of the histogram to obtain histogram calculation parameters, and adjusting and determining the fusion weight calculation formula according to the histogram calculation parameters;
[0057] S2: Obtaining brightness information in the original image, obtaining a downsampled image according to the brightness information and the original image, and obtaining a downsampled fusion weight according to the pixel brightness of the downsampled image and the fusion weight calculation formula;
[0058] S3: obtaining an upsampling fusion weight according to the downsampling fusion weight, so that the resolution of the image corresponding to the upsampling fusion weight is consistent with the resolution of the original image;
[0059] S4: Obtain a fused image according to the up-sampling fusion weight and the original image.
[0060] The advantages of the image fusion method of the present invention are as follows: by acquiring the histogram of the original image and the histogram calculation parameters, when image fusion is performed in different scenes, the fusion weight calculation formula is adjusted and determined by the histogram calculation parameters, and the statistical information of the histogram is fully utilized, so that the fusion weight calculation formula has higher adaptability, thereby improving the applicability of the image fusion method of the present invention, effectively preventing the problem of unsatisfactory fusion weight calculation caused by different relative performances in different scenes or the same brightness value, thereby improving the problems of highlight overexposure, dark information loss, and noise abnormality caused by it; the upsampling fusion weight of the original resolution is upsampled according to the downsampling weight calculated by the downsampling image, which reflects the overall information of the image, and the fused image is obtained by the upsampling fusion weight and the original image, thereby obtaining the local and overall information of the image, combining the local information and the overall information of the image, achieving the effect of maintaining the image highlights and dark details, reducing noise, and natural image transition, and improving the quality of the fused image; and solving the problem that the introduction of calculation in the HDR image fusion process may cause the loss of highlights or dark details and unnatural transition.
[0061] In some embodiments, the fused image finally acquired by the present invention is a high dynamic range image.
[0062] As an optional implementation manner of the present invention, in step S0, the step of obtaining the histogram of the original image includes:
[0063] In several different scenes, the original images in each scene at different exposure times are obtained respectively, and the histogram of the original image is obtained. The beneficial effect is that the histogram of the original image at different exposure times in different scenes is obtained, so that the image fusion method is applicable to a variety of different scenes, thereby improving the applicability of the image fusion method of the present invention.
[0064] As an optional implementation manner of the present invention, in step S1, the histogram calculation parameters include the median, integral and mean of the histogram.
[0065] In some embodiments, the histogram is a grayscale histogram.
[0066] As an optional implementation of the present invention, in step S1, the fusion weight calculation formula includes a weight calculation curve, and the step of adjusting and determining the fusion weight calculation formula according to the histogram calculation parameters includes:
[0067] A preset weight curve, wherein a first slope of the weight curve in a first defined interval is less than or equal to 0 and is monotonically decreasing, a second slope of the weight curve in a second defined interval is less than or equal to 0 and is monotonically increasing, and a right endpoint of the first defined interval is connected to a left endpoint of the second defined interval;
[0068] Record the point where the first slope changes from 0 to non-0 as the first inflection point, and record the point where the second slope changes from non-0 to 0 as the second inflection point;
[0069] The shape of the weight curve, the first slope, the second slope, at least one of the coordinates of the first inflection point and the coordinates of the second inflection point are adjusted according to the histogram calculation parameters to obtain the weight calculation curve. The advantage is that the shape of the weight curve, the first slope, the second slope, at least one of the coordinates of the first inflection point and the coordinates of the second inflection point are adjusted according to the histogram calculation parameters to obtain the weight calculation curve, so that the weight calculation curve is suitable for weight calculation of the original image corresponding to the histogram.
[0070] As an optional implementation manner of the present invention, in step S2, the step of acquiring a downsampled image according to the brightness information and the original image includes:
[0071] The original image is divided into m×n blocks, where m and n are both positive integers, the average brightness of all pixels in each block is calculated, and the brightness of all pixels in each block is replaced by the average brightness corresponding to the block to obtain the downsampled image with a resolution of m×n; each original image is subjected to the above steps to obtain a corresponding downsampled image with a resolution of m×n.
[0072] As an optional implementation of the present invention, in step S3, the down-sampling fusion weight includes a first exposure image weight and a second exposure image weight, and the step of obtaining the down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula includes:
[0073] Traversing all pixels of the downsampled image to obtain brightness of the pixels, inputting the brightness of the pixels into the weight calculation curve to obtain a first weight result, normalizing the first weight result to obtain the first exposure image weight and the second exposure image weight, wherein the sum of the first exposure image weight and the second exposure image weight is 1;
[0074] The first exposure image weight corresponds to the first exposure image, the second exposure image weight corresponds to the second exposure image, and the exposure time of the first exposure image is greater than the exposure time of the second exposure image.
[0075] Figure 2 is a schematic diagram of a weight curve according to an embodiment of the present invention, Figure 2 The ordinate of the weight curve in is the weight, and the abscissa is the brightness.
[0076] In some specific embodiments of the present invention, the step of acquiring a first fused image according to the downsampled image and the downsampled fusion weight comprises:
[0077] Reference Figure 2 , a preset weight curve f(G), wherein a first slope of the weight curve f(G) in a first defined interval [0, a] is less than or equal to 0 and is monotonically decreasing, and a second slope of the weight curve f(G) in a second defined interval [a, b] is less than or equal to 0 and is monotonically increasing;
[0078] In some embodiments, the weight curve is expressed as:
[0079]
[0080] Among them, G is the abscissa of the curve, specifically representing the brightness of the image, and f(G) is the ordinate of the curve.
[0081] Record the point where the first slope of the weight curve f(G) changes from 0 to non-0 on the first definition interval [0, a] as the first inflection point A, record the point where the second slope of the weight curve f(G) changes from non-0 to 0 on the second definition interval [a, b] as the second inflection point B, and adjust the shape of the weight curve, the first slope, the second slope, the coordinates of the first inflection point A and the coordinates of the second inflection point B according to the histogram calculation parameters, that is, adjust the shape of the weight curve, the first slope, the second slope, the coordinates of the first inflection point A and the coordinates of the second inflection point B by the histogram calculation parameters, so as to obtain the weight calculation curve w=f'(G);
[0082] Specifically, the information of the original image is converted into G channel brightness information, and the image is divided into m×n blocks, the average brightness of all pixels in each block is calculated, and the average brightness of the pixels in the block is used as the brightness of the block to obtain a downsampled image with a resolution of m×n;
[0083] Substituting the brightness of the sampled image into the weight calculation curve w=f'(G) to obtain a first weight result, normalizing the first weight result to obtain the first exposure image weight w and the second exposure image weight (1-w), wherein the sum of the first exposure image weight w and the second exposure image weight (1-w) is 1;
[0084] The first exposure image weight w corresponds to the first exposure image l1, the second exposure image weight (1-w) corresponds to the second exposure image l2, and the exposure time of the first exposure image l1 is greater than the exposure time of the second exposure image l2.
[0085] As an optional implementation of the present invention, in step S1, the fusion weight calculation formula includes a one-dimensional weight calculation Gaussian function, and the step of adjusting and determining the calculation formula of the fusion weight according to the histogram calculation parameters includes:
[0086] A one-dimensional Gaussian function is preset, and the one-dimensional Gaussian function is adjusted according to the histogram calculation parameters to obtain the one-dimensional weight calculation Gaussian function. The advantage is that by presetting the one-dimensional Gaussian function and adjusting the one-dimensional Gaussian function according to the histogram calculation parameters, the one-dimensional weight calculation Gaussian function obtained can be better suitable for the weight calculation of the original image corresponding to the histogram.
[0087] As an optional implementation of the present invention, in step S3, the step of obtaining the down-sampling fusion weight according to the pixel brightness of the down-sampled image and the fusion weight calculation formula includes:
[0088] The brightness of each pixel in the downsampled image is input into the one-dimensional weight calculation Gaussian function to obtain a second weight result, and the second weight result is normalized to obtain the downsampled fusion weight, wherein the downsampled fusion weight includes N weights, where N is a positive integer.
[0089] In some other specific embodiments of the present invention, the step of acquiring a first fused image according to the downsampled image and the downsampled fusion weight includes:
[0090] Obtaining a histogram of the original image through the original image, and extracting features of the histogram to obtain histogram calculation parameters;
[0091] A one-dimensional Gaussian function is preset, and the formula of the one-dimensional Gaussian function is:
[0092]
[0093] Wherein, x represents the input original image information, in the embodiment of the present invention, x refers to brightness, σ is the Gaussian standard deviation, and μ represents the optimal pixel value;
[0094] Adjusting the one-dimensional Gaussian function according to the histogram calculation parameters, specifically, adjusting the Gaussian standard deviation σ and the optimal pixel value μ by the histogram calculation parameters to obtain the one-dimensional weighted Gaussian function w=g'(x);
[0095] The brightness of each pixel in the downsampled image is input into the one-dimensional weight calculation Gaussian function w=g'(x) to obtain a second weight result, and the second weight result is normalized to obtain the downsampled fusion weight, wherein the downsampled fusion weight includes N weights, where N is a positive integer, and the N weights are respectively a first weight, a second weight, ..., an Nth weight, and the weighted sum of the N weights is 1.
[0096] In some embodiments, the weight of the Gaussian standard deviation σ and the optimal pixel value μ is determined based on the number of points in the histogram that are greater than a certain value, and then the values of the Gaussian standard deviation σ and the optimal pixel value μ are adjusted, so that the one-dimensional weight calculation Gaussian function finally obtained is more suitable for the weight calculation of the image in the current scene.
[0097] In some specific embodiments, the Gaussian standard deviation σ is 0.5, and the optimal pixel value μ is a weighted average value of different frame images.
[0098] The present invention obtains the weight calculation curve by adjusting the weight curve of the exposure image, and obtains the one-dimensional weight calculation Gaussian function by adjusting the one-dimensional Gaussian function. The weight of the image can be calculated by any one of the weight calculation curve and the one-dimensional weight calculation Gaussian function, thereby providing a variety of methods and calculation approaches for calculating the fusion weight, thereby improving the applicability and practicality of the image fusion method of the present invention, and can be applied to a variety of scenarios.
[0099] It can be explained that, for the processing of the same group of original images, only one of the weight calculation curve and the one-dimensional weight calculation Gaussian function is used to calculate the weight of the image, that is, in the same group of original images, the weight calculation curve and the one-dimensional weight calculation Gaussian function cannot be used simultaneously to calculate the image weight, so as to reduce the error of weight calculation and avoid the problem of unnatural image fusion caused by using different weight calculation methods.
[0100] As an optional implementation manner of the present invention, obtaining the upsampling fusion weight according to the downsampling fusion weight includes the steps of:
[0101] The down-sampled fusion weight is up-sampled by bilinear interpolation to obtain the up-sampled fusion weight.
[0102] In some specific embodiments, the step of performing upsampling processing on the downsampled fusion weights by bilinear interpolation to obtain the upsampled fusion weights includes:
[0103] Figure 3 FIG. 4 is a schematic diagram of a bilinear interpolation method according to an embodiment of the present invention.
[0104] In some specific embodiments, referring to Figure 3 , use the down-sampling fusion weights of the four neighboring pixels of the pixel to be determined to perform linear interpolation in two directions. The coordinates of the four points Q11, Q12, Q21, and Q22 of the known function f are Q11(x1, y1), Q12(x1, y2), Q21(x2, y1), and Q22(x2, y2) in sequence.
[0105] First, perform linear interpolation in the x direction, insert R2 into Q12 and Q22, and insert point R1 into Q11 and Q21 to obtain R1 and R2;
[0106] Then perform linear interpolation in the y direction, and interpolate the R1 and R2 calculated in the first step in the y direction to calculate point P, and get f(x,y). The calculation formula of f(x,y) is as follows:
[0107]
[0108] The coordinates (x, y) of the final interpolation point P are obtained by the above formula, and the four points in each downsampled image are combined to obtain the interpolation point of each downsampled image.
[0109] Figure 4 Schematic diagram of a grid of a bilinear interpolation method according to an embodiment of the present invention. The numbers on the z-axis in the figure represent weight values of points, and the numbers on the x-axis and y-axis represent the horizontal and vertical coordinates of the points, respectively.
[0110] In some specific embodiments, the calculation formula of f(x, y) is as follows: Figure 4 The grid diagram of the bilinear interpolation method is shown in FIG. The corresponding interpolation weights of the current point at each coordinate in the upsampled image are calculated, that is, the coordinates of the interpolation point P are substituted into Figure 4 The grid diagram of the bilinear interpolation method shown can obtain the weight of the interpolation point P, and the upsampling weight is calculated based on the weight of the interpolation point P; the upsampling weights of the pixels at the same position in several images are normalized to obtain the upsampling fusion weight; a first weight gradient map is obtained based on the upsampling fusion weight, and the resolution of the weight gradient map is consistent with the resolution of the original image.
[0111] As an optional implementation manner of the present invention, obtaining the upsampling fusion weight according to the downsampling fusion weight includes the steps of:
[0112] The down-sampled fusion weight is up-sampled by a two-dimensional Gaussian function to obtain the up-sampled fusion weight.
[0113] In some specific embodiments, the step of performing upsampling processing on the downsampled fusion weights by using a two-dimensional Gaussian function to obtain the upsampled fusion weights includes:
[0114] Preset two-dimensional Gaussian function, the formula of two-dimensional Gaussian function is:
[0115]
[0116] Here, σ0 is the Gaussian standard deviation.
[0117] Figure 5 is a surface diagram formed by a two-dimensional Gaussian function according to an embodiment of the present invention, Figure 5 The numbers on the Z-axis represent the weight values.
[0118] In some embodiments, reference Figure 5 , from the above two-dimensional Gaussian function, we can get (x center ,y center ) as the center, For each point in the downsampled image, the weight values of the current downsampled image B(X,Y) and the surrounding blocks B(X-1,Y-1), B(X,Y-1)...B(X+1,Y+1) and the function value f(x',y') of the corresponding coordinates (x',y') of the surface are combined to obtain a second weight gradient map radiating from the center to the surroundings, and the resolution of the second weight gradient map is consistent with the resolution of the original image.
[0119] Reference Figure 5 , for each point in the downsampled image, the coordinates of the left B(X,Y) and surrounding blocks B(X-1,Y-1), B(X,Y-1)...B(X+1,Y+1) of the current point in the current downsampled image are substituted into Figure 5 The surface diagram formed by the two-dimensional Gaussian function shown in the figure is used to calculate the corresponding interpolation weights of the current point at each coordinate in the upsampled image, and the upsampling weight of the current point is calculated based on the interpolation weight; the upsampling weights of the pixel points at the same position in several images are normalized to obtain the upsampling fusion weight.
[0120] In some embodiments, in step S4, the step of obtaining a one-dimensional weighted Gaussian function image according to the up-sampled fusion weight and the original image comprises:
[0121] After upsampling the downsampling fusion weights by the above-mentioned bilinear interpolation method or two-dimensional Gaussian function to obtain the upsampling fusion weights, after obtaining the upsampling fusion weights of several original images, the fused image is obtained by weighted fusion. The specific formula is as follows:
[0122] ImgHDR=w1×Img1+w2×Img2+…+wN×ImgN
[0123] Among them, ImgHDR is the fused image, Img1 is the first original image, Img2 is the second original image,…, ImgN is the Nth original image, and N is a positive integer; w1 is the upsampling fusion weight corresponding to the first original image Img1, w2 is the upsampling fusion weight corresponding to the second original image Img2,…, wN is the upsampling fusion weight corresponding to the Nth original image ImN, w1+w2+…+wN=1.
[0124] The embodiment of the present invention adopts bilinear interpolation and two-dimensional Gaussian function to upsample the downsampled fusion weights to obtain the upsampled fusion weights, which provides a variety of calculation paths for calculating the upsampled fusion weights, thereby improving the applicability of the image fusion method described in the present invention, saving algorithm steps, and improving the upsampling efficiency.
[0125] It can be explained that, for the processing of the same group of original images, only one of the bilinear interpolation method and the two-dimensional Gaussian function is used to calculate the upsampling fusion weight for the same group of original images, that is, the bilinear interpolation method and the two-dimensional Gaussian function cannot be used simultaneously to calculate the upsampling fusion weight in the same group of original images, so as to reduce the error of weight calculation and avoid the problem of unnatural image fusion caused by using different weight calculation methods.
[0126] The present invention also provides an image fusion device. Figure 6 It is a structural block diagram of the image fusion device of the present invention.
[0127] In some embodiments, reference Figure 6 , the image fusion device comprises:
[0128] An image processing module 1 is used to provide an original image, obtain a histogram of the original image and brightness information in the original image, and obtain a downsampled image based on the brightness information and the original image;
[0129] An adjustment module 2 is used to extract the features of the histogram to obtain histogram calculation parameters, and adjust and determine the fusion weight calculation formula according to the histogram calculation parameters;
[0130] A weight calculation module 3, used to obtain a down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula, and to obtain an up-sampling fusion weight according to the down-sampling fusion weight, so that the resolution of the image corresponding to the up-sampling fusion weight is consistent with the resolution of the original image;
[0131] The image fusion calculation module 4 is used to obtain a fused image according to the up-sampling fusion weight and the original image.
[0132] The advantages of the image fusion device of the present invention are: the histogram and brightness information of the original image are obtained through the image processing module 1, and the downsampled image is obtained according to the brightness information and the original image, the histogram calculation parameters are obtained through the adjustment module 2, and the fusion weight calculation formula is adjusted and determined according to the histogram calculation parameters, the downsampled fusion weight and the upsampled fusion weight are obtained through the weight calculation module 3, and the image fusion calculation module 4 obtains the fused image according to the upsampled fusion weight and the original image; the statistical information of the histogram is fully utilized, the applicability of the fusion weight calculation formula is improved, and the applicable scenarios of the image fusion method of the present invention are expanded; by calculating the downsampled image, the downsampled fusion weight and the upsampled fusion weight, the overall image and the local image are combined, so as to achieve the effect of maintaining the image highlights and dark details, reducing noise, and natural image transition, thereby improving the quality of the fused image; and the problem that the introduction of calculation in the HDR image fusion process may cause the loss of highlights or dark details and unnatural transition is solved.
[0133] In some embodiments, reference Figure 6 The step of obtaining the down-sampling fusion weight according to the pixel brightness of the down-sampled image and the fusion weight calculation formula comprises:
[0134] The histogram of the original image in each scene at different exposure times is obtained in a plurality of different scenes by the image processing module 1;
[0135] A weight curve is preset by adjusting module 2, wherein a first slope of the weight curve in a first defined interval is less than or equal to 0 and is monotonically decreasing, and a second slope of the weight curve in a second defined interval is less than or equal to 0 and is monotonically increasing, a right endpoint of the first defined interval is connected to a left endpoint of the second defined interval, a point where the first slope changes from 0 to non-0 is recorded as a first inflection point, and a point where the second slope changes from non-0 to 0 is recorded as a second inflection point, and a shape of the weight curve, the first slope, the second slope, at least one of the coordinates of the first inflection point and the coordinates of the second inflection point are adjusted according to the histogram calculation parameters to obtain the weight calculation curve;
[0136] The original image is divided into m×n blocks by the image processing module 1, where m and n are both positive integers, the average brightness of all pixels in each block is calculated, and the brightness of all pixels in each block is replaced by the average brightness corresponding to the block to obtain the downsampled image with a resolution of m×n;
[0137] The weight calculation module 3 traverses all pixel points of the downsampled image to obtain the brightness of the pixel points, and inputs the brightness of the pixel points into the weight calculation curve to obtain a first weight result, and the first weight result is normalized to obtain the first exposure image weight and the second exposure image weight, and the sum of the first exposure image weight and the second exposure image weight is 1.
[0138] In other embodiments of the present invention, referring to Figure 6 The step of obtaining the down-sampling fusion weight according to the pixel brightness of the down-sampled image and the fusion weight calculation formula comprises:
[0139] Presetting a one-dimensional Gaussian function through the adjustment module 2, and adjusting the one-dimensional Gaussian function according to the histogram calculation parameter to obtain the one-dimensional weight calculation Gaussian function;
[0140] The image processing module 1 divides the original image into m×n blocks, where m and n are both positive integers, and each original image obtains a corresponding down-sampled image with a resolution of m×n;
[0141] The brightness of each pixel in the downsampled image is input into the one-dimensional weight calculation Gaussian function through the weight calculation module 3 to obtain a second weight result, and the second weight result is normalized to obtain a first weight, a second weight, ..., an Nth weight, where N is a positive integer.
[0142] In some embodiments of the present invention, reference Figure 6 The weight calculation module 3 performs upsampling processing on the down-sampling fusion weight through a bilinear interpolation method to obtain the up-sampling fusion weight.
[0143] In other embodiments of the present invention, referring to Figure 6 The weight calculation module 3 performs upsampling processing on the down-sampling fusion weight through a two-dimensional Gaussian function to obtain the up-sampling fusion weight.
[0144] The present invention also provides a device, comprising a memory, a processor and a program stored in the memory and executable on the processor, wherein the image fusion method is implemented when the processor executes the program.
[0145] The present invention also provides a storage medium on which a program is stored, and when the program is executed by a processor, the image fusion method is implemented.
[0146] The advantages of the device and storage medium of the present invention are as follows: by adjusting and determining the fusion weight calculation formula through the histogram, the statistical information of the histogram is fully utilized, the applicability of the fusion weight calculation formula is improved, and the applicable scenarios of the image fusion method of the present invention are expanded. By calculating the downsampled image, the downsampled fusion weight and the upsampled fusion weight, the overall image and the local image are combined, so as to achieve the effects of maintaining the image highlights and dark details, reducing noise, and natural image transition, thereby improving the quality of the fused image and solving the problem that the introduction of calculation in the HDR image fusion process may cause the loss of highlights or dark details and unnatural transition.
[0147] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.
Claims
1. An image fusion method, characterized in that: Includes steps: Providing an original image, and obtaining a histogram of the original image; Extract the features of the histogram to obtain histogram calculation parameters, adjust and determine the fusion weight calculation formula according to the histogram calculation parameters, wherein the fusion weight calculation formula includes a weight calculation curve, and the steps of adjusting and determining the fusion weight calculation formula according to the histogram calculation parameters include: presetting a weight curve, wherein a first slope of the weight curve in a first definition interval is less than or equal to 0 and is monotonically decreasing, and a second slope of the weight curve in a second definition interval is less than or equal to 0 and is monotonically increasing, and the right endpoint of the first definition interval is connected to the left endpoint of the second definition interval; record the point where the first slope changes from 0 to non-0 as the first inflection point, and record the point where the second slope changes from non-0 to 0 as the second inflection point; adjust the shape of the weight curve, the first slope, the second slope, the coordinates of the first inflection point, and the coordinates of the second inflection point according to the histogram calculation parameters to obtain the weight calculation curve; Acquire brightness information in the original image, acquire a downsampled image according to the brightness information and the original image, and acquire a downsampled fusion weight according to the pixel brightness of the downsampled image and the fusion weight calculation formula; Obtaining an upsampling fusion weight according to the downsampling fusion weight, so that a resolution of an image corresponding to the upsampling fusion weight is consistent with a resolution of the original image; A fused image is obtained according to the up-sampling fusion weight and the original image.
2. The image fusion method according to claim 1, characterized in that: The step of obtaining the histogram of the original image comprises: In a number of different scenes, original images in each of the scenes at different exposure times are acquired respectively, and a histogram of the original image is acquired.
3. The image fusion method according to claim 1, characterized in that: The histogram calculation parameters include the median, integral and mean of the histogram.
4. The image fusion method according to claim 1, characterized in that: The down-sampling fusion weight includes a first exposure image weight and a second exposure image weight. The step of obtaining the down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula includes: Traversing all pixels of the downsampled image to obtain brightness of the pixels, inputting the brightness of the pixels into the weight calculation curve to obtain a first weight result, normalizing the first weight result to obtain the first exposure image weight and the second exposure image weight, wherein the sum of the first exposure image weight and the second exposure image weight is 1; The first exposure image weight corresponds to the first exposure image, the second exposure image weight corresponds to the second exposure image, and the exposure time of the first exposure image is greater than the exposure time of the second exposure image.
5. The image fusion method according to claim 1, characterized in that: The fusion weight calculation formula includes a one-dimensional weight calculation Gaussian function, and the steps of adjusting and determining the fusion weight calculation formula according to the histogram calculation parameters include: A one-dimensional Gaussian function is preset, and the one-dimensional Gaussian function is adjusted according to the histogram calculation parameter to obtain the one-dimensional weight calculation Gaussian function.
6. The image fusion method according to claim 5, characterized in that: The step of obtaining the down-sampling fusion weight according to the pixel brightness of the down-sampled image and the fusion weight calculation formula comprises: The brightness of each pixel in the downsampled image is input into the one-dimensional weight calculation Gaussian function to obtain a second weight result, and the second weight result is normalized to obtain the downsampled fusion weight, wherein the downsampled fusion weight includes N weights, where N is a positive integer.
7. The image fusion method according to claim 1, characterized in that: The step of obtaining a downsampled image according to the brightness information and the original image comprises: The original image is divided into m×n blocks, where m and n are both positive integers, the average brightness of all pixels in each block is calculated, and the brightness of all pixels in each block is replaced by the average brightness corresponding to the block to obtain the downsampled image with a resolution of m×n.
8. The image fusion method according to claim 1, characterized in that: Obtaining the upsampling fusion weight according to the downsampling fusion weight comprises the following steps: The down-sampled fusion weight is up-sampled by bilinear interpolation to obtain the up-sampled fusion weight.
9. The image fusion method according to claim 1, characterized in that: Obtaining the upsampling fusion weight according to the downsampling fusion weight comprises the following steps: The down-sampled fusion weight is up-sampled by a two-dimensional Gaussian function to obtain the up-sampled fusion weight.
10. An image fusion device, used to implement the image fusion method according to any one of claims 1 to 9, characterized in that: The image fusion device comprises: An image processing module, used to provide an original image, obtain a histogram and brightness information of the original image, and obtain a downsampled image based on the brightness information and the original image; An adjustment module, used for extracting the features of the histogram to obtain histogram calculation parameters, and adjusting and determining a fusion weight calculation formula according to the histogram calculation parameters; A weight calculation module, used to obtain a down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula, and to obtain an up-sampling fusion weight according to the down-sampling fusion weight, so that the resolution of the image corresponding to the up-sampling fusion weight is consistent with the resolution of the original image; The image fusion calculation module is used to obtain a fused image according to the up-sampling fusion weight and the original image.
11. A device comprising a memory, a processor and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the image fusion method according to any one of claims 1 to 9 is implemented.
12. A storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the image fusion method according to any one of claims 1 to 9 is implemented.
Citation Information
Patent Citations
Multi-frame HDR image processing method and device, storage medium and electronic equipment
CN111462031A
Super-resolution image reconstruction method and device, medium, and equipment
CN108154474A
Multi-exposure image fusion method
CN109636767A
Image fusion method and device, terminal equipment and storage medium
CN112837254A
Image processing method, image processing device, terminal and readable storage medium
CN113313645A
