Hybrid exposure imaging method and apparatus for high dynamic range

Abstract

The application provides a high dynamic range hybrid exposure imaging method and device, applied to the technical field of image processing, and includes the following steps: acquiring a plurality of hybrid exposure images; respectively performing noise reduction on normal exposure images and long exposure images to obtain noise-reduced normal exposure images and noise-reduced long exposure images; performing repair on an ultra-short exposure image, the noise-reduced normal exposure images and the noise-reduced long exposure images to obtain a repaired ultra-short exposure image, a repaired normal exposure image and a repaired long exposure image; performing weighted fusion on the repaired ultra-short exposure image, a short exposure image, the repaired normal exposure image and the repaired long exposure image to obtain a fusion image; and performing noise reduction on the fusion image according to the brightness of the fusion image to obtain a high dynamic range image. The clarity of the image is improved by respectively performing noise reduction and repair on different exposure images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a high dynamic range hybrid exposure imaging method and device. BACKGROUND

[0002] At present, a dynamic range image can be collected through an image sensor. However, the bit width of a conventional image sensor is small, and the dynamic range of an image collected by one exposure is limited, while the dynamic range of the nature is much higher than the dynamic range that can be collected by the image sensor, which leads to the situation that the image collected by the image sensor may be too dark in the dark part or overexposed in the bright part in a high dynamic range scene.

[0003] At present, a high dynamic range image can be obtained by surrounding exposure, specifically, a plurality of collected images can be synthesized into a high dynamic range image by means of a pre-designed algorithm. However, it is found in practice that the above imaging method usually uses a uniform noise reduction strength to reduce the noise of the image. Since the noise levels of different regions in the image are different, using the same noise reduction strength to reduce the noise of regions with different noise levels may cause the problem of over-reducing the noise of some regions and under-reducing the noise of some regions, thereby resulting in a low definition of the obtained image.

[0004] Therefore, a new high dynamic range imaging technical solution is needed. SUMMARY

[0005] Therefore, the embodiments of the present application provide a high dynamic range hybrid exposure imaging method and device, which can solve the problem of over-reducing the noise of some regions or under-reducing the noise of some regions in the image, thereby improving the definition of the image.

[0006] The embodiments of the present application provide the following technical solutions:

[0007] In a first aspect, the embodiments of the present application provide a high dynamic range hybrid exposure imaging method, which comprises:

[0008] obtaining a plurality of hybrid exposure images; wherein the plurality of hybrid exposure images at least include an ultra-short exposure image, a short exposure image, a normal exposure image and a long exposure image;

[0009] respectively reducing the noise of the normal exposure image and the long exposure image to obtain a noise-reduced normal exposure image and a noise-reduced long exposure image;

[0010] repairing the ultra-short exposure image, the noise-reduced normal exposure image and the noise-reduced long exposure image to obtain a repaired ultra-short exposure image, a repaired normal exposure image and a repaired long exposure image;

[0011] perform weighted fusion on the patched ultra-short exposure image, the short exposure image, the patched normal exposure image and the patched long exposure image to obtain a fusion image;

[0012] perform noise reduction on the fusion image according to brightness of the fusion image to obtain a high dynamic range image.

[0013] In a second aspect, the embodiments of the present specification also provide a high dynamic range hybrid exposure imaging device, the device comprising:

[0014] an acquisition unit configured to acquire a plurality of hybrid exposure images; wherein the plurality of hybrid exposure images at least include an ultra-short exposure image, a short exposure image, a normal exposure image and a long exposure image;

[0015] a first noise reduction unit configured to perform noise reduction on the normal exposure image and the long exposure image respectively to obtain a noise-reduced normal exposure image and a noise-reduced long exposure image;

[0016] a patching unit configured to patch the ultra-short exposure image, the noise-reduced normal exposure image and the noise-reduced long exposure image to obtain a patched ultra-short exposure image, a patched normal exposure image and a patched long exposure image;

[0017] a fusion unit configured to perform weighted fusion on the patched ultra-short exposure image, the short exposure image, the patched normal exposure image and the patched long exposure image to obtain a fusion image;

[0018] a second noise reduction unit configured to perform noise reduction on the fusion image according to brightness of the fusion image to obtain a high dynamic range image.

[0019] In a third aspect, the embodiments of the present specification also provide a computer readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of the first aspect.

[0020] In a fourth aspect, the embodiments of the present specification also provide a computing device, the computing device comprising:

[0021] at least one processor, a memory and an input output unit;

[0022] wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program stored in the memory to perform the method of any one of the first aspect.

[0023] Compared with the prior art, the at least one technical solution adopted by the embodiment of the present specification can achieve the beneficial effects at least including: by respectively performing noise reduction and repairing on different exposure images, actual conditions of different regions in the image can be respectively reduced, the problems of over-reduction or insufficient reduction in some regions can be avoided, and the clarity of the image is improved. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0025] Figure 1 is a flowchart of a high dynamic range hybrid exposure imaging method in the present application;

[0026] Figure 2 is a flowchart of a hybrid exposure multi-frame image acquisition method in the present application;

[0027] Figure 3 is a flowchart of a noise reduction method based on normal exposure images and long exposure images in the present application;

[0028] Figure 4 is a flowchart of a repair method based on hybrid exposure multi-frame images in the present application;

[0029] Figure 5 is a flowchart of a noise reduction method based on fusion images in the present application;

[0030] Figure 6 is a structural diagram of a high dynamic range hybrid exposure imaging device in the present application;

[0031] Figure 7 is a structural diagram of a medium in the present application;

[0032] Figure 8 is a structural diagram of a computing device in the present application. DETAILED DESCRIPTION

[0033] The embodiments of the present application will be described in detail below with reference to the drawings.

[0034] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0036] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0037] Additionally, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that practice can be carried out without these specific details.

[0038] The technical solutions provided by the various embodiments of this application are described below with reference to the accompanying drawings.

[0039] like Figure 1 As shown in the embodiments of this specification, a high dynamic range hybrid exposure imaging method is provided, which may include:

[0040] Step S101: Obtain multi-frame images with mixed exposure.

[0041] In the embodiment of the present application, the multi-frame image of mixed exposure includes at least an ultra-short exposure image, a short exposure image, a normal exposure image and a long exposure image. The multi-frame image of mixed exposure is collected by an image collection device, which can be a camera, a video camera or the like. The exposure values of the ultra-short exposure image, the short exposure image, the normal exposure image and the long exposure image are different when the image collection device is collecting images. The exposure value can be a fixed value set in advance, or a value matched with the environment of the image collection device.

[0042] For example, the exposure value corresponding to the normal exposure image can be the automatic exposure value (AEV) of the image collection device, the exposure value corresponding to the short exposure image can be half of the exposure value corresponding to the normal exposure image, the exposure value corresponding to the ultra-short exposure image can be half of the exposure value corresponding to the short exposure image, and the exposure value of the long exposure image can be twice the exposure value corresponding to the normal exposure image.

[0043] In another embodiment of the present application, in order to ensure that the multi-frame image of mixed exposure is more suitable for generating a high dynamic range image, different exposure strategies can be set for different environments, such as Figure 2 As shown in FIG. 1, the step S101 is replaced by the following steps S201-S203:

[0044] Step S201: acquiring a current environment in which the image collection device is located.

[0045] Step S202: determining an exposure strategy matched with the current environment.

[0046] Step S203: acquiring a multi-frame image of mixed exposure according to the exposure strategy.

[0047] The implementation of the above steps S201-S203 ensures that the multi-frame image of mixed exposure is more suitable for generating a high dynamic range image, because the light and the dynamic range are different in different environments, and thus different exposure strategies need to be set for different environments.

[0048] As an optional implementation, before the step S202 of determining the exposure strategy matched with the current environment, the method can further include the following steps: acquiring a training data set, the training data set including at least environment information and an environment category corresponding thereto; training a scene classification model based on the training data set to obtain a trained scene classification model.

[0049] In addition, the way of determining the exposure strategy matched with the current environment in the step S202 can be: inputting the current environment into the scene classification model to obtain the exposure strategy matched with the current environment.

[0050] According to the embodiment, the scene classification model can be trained according to a large amount of training data sets obtained, so that the scene classification model can accurately output an accurate exposure strategy according to input environmental information.

[0051] In the embodiment, the scene classification model can be constructed using the mobilenet-v3, and then trained on the place365 data set, and then migrated learning is performed according to an actual use scene to obtain the trained scene classification model.

[0052] In step S102, the normal exposure image and the long exposure image are respectively denoised to obtain a denoised normal exposure image and a denoised long exposure image.

[0053] In the embodiment, the number of the normal exposure images and the long exposure images is multiple.

[0054] For example, one frame of ultra-short exposure image, one frame of short exposure image, four frames of normal exposure image and four frames of long exposure image can be collected, and the embodiment is not limited thereto. In the final high dynamic range synthesis image, the pixel value of the low-light area is from the normal exposure image and the long exposure image collected by the image, and the image signal is usually generated by the voltage imaging of the charge accumulated by the photoelectric effect of the thermal radiation photoelectric effect and the photoelectric effect of the photons entering the lens. However, for the low-light area, the number of photons entering the lens is small, and the main component is the electron excited by thermal radiation, so the signal-to-noise ratio of the low-light area is usually low. In order to improve the signal-to-noise ratio of the low-light part, multiple frames of normal exposure and long exposure are collected for time domain superposition and denoising to improve the signal-to-noise ratio of the low-light area. According to the central limit theorem analysis, the noise variance of the synthesis image obtained by using four frames of images is reduced to one fourth of the original, which can significantly improve the signal-to-noise ratio of the low-light area.

[0055] In another embodiment, in order to effectively improve the signal-to-noise ratio of the normal exposure image and the long exposure image dark image signal, multiple frames of the normal exposure image and the long exposure image can be superimposed and denoised, as shown in Figure 3 The step S102 is replaced by the following steps S301-S307:

[0056] In step S301, a reference normal exposure image is selected from the multiple frames of the normal exposure image.

[0057] In step S302, a reference long exposure image is selected from the multiple frames of the long exposure image.

[0058] Step S303, determining a first homography matrix of normal exposure images other than the reference normal exposure image in the plurality of normal exposure images according to the reference normal exposure image.

[0059] As an optional implementation, the manner of step S303 of determining the first homography matrix of the normal exposure images other than the reference normal exposure image in the plurality of normal exposure images according to the reference normal exposure image can include the following steps:

[0060] determining the ORB features of the reference normal exposure image and a target normal exposure image other than the reference normal exposure image in the plurality of normal exposure images;

[0061] determining the first homography matrix of each target normal exposure image according to the ORB features of the reference normal exposure image and the ORB features of each target normal exposure image.

[0062] In this implementation, the homography matrix describes the global spatial position relationship between two images, including translation, rotation, scaling, and affine relationship. Since the pixel region of the normal exposure image used for high dynamic range imaging is mainly the middle and low brightness region, we can assign higher weights to the pixel value error of the middle and low brightness region in the error formula of the homography matrix calculation, so that the error control of the middle and low brightness region of the normal exposure image is more strict.

[0063] In the embodiment of the application, one normal exposure image is selected as the reference normal exposure image I m0 The remaining normal exposure images can be I m1 , I m2 and I m3 The ORB features orb0, orb1, orb2 and orb3 of I m0 , I m1 , I m2 and I m3 can be calculated, and the RANSAC algorithm can be used to calculate the first homography matrix H 01 , H 02 and H 03 of I m0 and I m1 , I m2 and I m3 , respectively.

[0064]

[0065] In addition, in order to avoid that the ORB features are only gathered to a certain space in the image space, the normal exposure image can be divided by using an octree, and the ORB features of each region are calculated respectively, and finally used to cooperate with the global registration of the normal exposure image.

[0066] In step S304, a second homography matrix of the long exposure images other than the reference long exposure image in the plurality of long exposure images is determined according to the reference long exposure image.

[0067] As an optional embodiment, the manner of determining the second homography matrix of the long exposure images other than the reference long exposure image in the plurality of long exposure images according to the reference long exposure image in step S304 can include the following steps:

[0068] ORB features of the reference long exposure image and a target long exposure image other than the reference long exposure image in the plurality of long exposure images are determined.

[0069] The second homography matrix of each target long exposure image is determined according to the ORB features of the reference long exposure image and the ORB features of the target long exposure image.

[0070] In this embodiment, the homography matrix describes the global spatial position relationship between two images, including translation, rotation, scaling, and affine relationship. Since the pixel region of the long exposure image used for high dynamic range imaging is mainly the middle and low brightness region, the error of the middle and low brightness region pixel value can be given a higher weight in the error formula of the homography matrix calculation, so that the error control of the middle and low brightness region of the long exposure image is more strict.

[0071] In the embodiment of the application, the determination manner of the second homography matrix of the long exposure image is the same as the determination manner of the first homography matrix of the normal exposure image, which is not described herein.

[0072] In step S305, the normal exposure images other than the reference normal exposure image are registered based on the first homography matrix and the reference normal exposure image, to obtain first registered normal exposure images.

[0073] In the embodiment of the application, the bicubic interpolation algorithm can be used to perform global registration on the normal exposure images other than the reference normal exposure image, and the obtained first registered normal exposure images are matched in brightness with the reference normal exposure image.

[0074] In step S306, the long exposure images other than the reference long exposure image are registered based on the second homography matrix and the reference long exposure image, to obtain first registered long exposure images.

[0075] In the embodiment of the present application, bicubic interpolation algorithm can be used to globally register the long-exposure images other than the reference normal-exposure image, and the obtained first registered long-exposure image is matched with the reference normal-exposure image in brightness.

[0076] In step S307, spatial noise reduction is performed on the first motion region in the normal-exposure image, the first motion region in the first registered normal-exposure image, the second motion region in the long-exposure image, and the second motion region in the first registered long-exposure image, to obtain a noise-reduced normal-exposure image and a noise-reduced long-exposure image.

[0077] After the steps S301-S307 are implemented, since the normal-exposure image and the long-exposure image are usually overexposed in bright parts and have limited effective content, and the effective content is mainly concentrated in dark parts, multi-frame superposition noise reduction of the normal-exposure image and the long-exposure image can effectively improve the signal-to-noise ratio of the dark part image signals of the two images.

[0078] As an optional implementation, the manner of performing spatial noise reduction on the first motion region in the normal-exposure image, the first motion region in the first registered normal-exposure image, the second motion region in the long-exposure image, and the second motion region in the first registered long-exposure image to obtain a noise-reduced normal-exposure image and a noise-reduced long-exposure image in step S307 can include the following steps:

[0079] According to the pixel values of each pixel in the reference normal-exposure image and the pixel values of each pixel in the first registered normal-exposure image, a first sum of absolute differences of each pixel in the reference normal-exposure image is obtained;

[0080] According to the pixel values of each pixel in the reference long-exposure image and the pixel values of each pixel in the first registered long-exposure image, a second sum of absolute differences of each pixel in the reference long-exposure image is obtained;

[0081] The region represented by the pixels with the first sum of absolute differences in the reference normal-exposure image greater than or equal to a first preset threshold is determined as the first motion region;

[0082] The region corresponding to the first motion region in the first registered normal-exposure image is also determined as the first motion region;

[0083] The region represented by the pixels with the second sum of absolute differences in the reference long-exposure image greater than or equal to a first preset threshold is determined as the second motion region;

[0084] The region corresponding to the second motion region in the first registered long-exposure image is also determined as the second motion region.

[0085] The first motion region and the second motion region are subjected to spatial noise reduction to obtain a normal-exposure image after noise reduction and a long-exposure image after noise reduction.

[0086] In this embodiment, various local motions such as non-rigid deformation may exist in the normal-exposure image and the long-exposure image, so the motion regions of the local motions in the normal-exposure image and the long-exposure image are determined first, and then noise reduction is specially performed on the motion regions, so that the noise reduction effect on the normal-exposure image and the long-exposure image is better.

[0087] In the embodiment, after global registration, there are still a small amount of motion regions in the normal-exposure image and the long-exposure image, so time-domain image noise reduction needs to be performed on the motion regions.

[0088] Take the coordinates (x, y) in the reference normal-exposure image I m0 as an example, I' m1 , I' m2 , I' m2 may be the first registered normal-exposure image, a 5x5 region is selected based on the coordinates (x, y), and the summation of absolute difference (SAD) of the 5x5 region of the coordinates (x, y) in I m0 and the 5x5 region of the coordinates (x, y) in I' m1 , I' m2 , I' m2 is calculated.

[0089]

[0090] Then, whether the pixel point is in the local motion region is judged according to a given threshold value:

[0091]

[0092] wherein thresh is a first preset threshold value, if move(x, y) is 0, it indicates that the region where the coordinates (x, y) are located has no motion, otherwise, the region where the coordinates (x, y) are located has local motion.

[0093] If the region has no local motion, then time-domain superposition noise reduction is performed, and the calculation formula is:

[0094]

[0095] wherein when k = 0, I' mk = Im0 ;

[0096] If there is local motion in a region, then spatial denoising can be used to reduce the noise level to be consistent with the noise level without the local region, and the spatial denoising can use a 5x5 region non-local means algorithm.

[0097] In the embodiment of the application, after the first registered normal exposure image and the first registered long exposure image are denoised, a denoised normal exposure image and a denoised long exposure image can be obtained; the signal-to-noise ratio of the denoised normal exposure image and the denoised long exposure image in the medium-low light region is greatly improved, which will greatly improve the dark image quality of the final high dynamic range imaging image.

[0098] In step S103, the super-short exposure image, the denoised normal exposure image and the denoised long exposure image are repaired to obtain a repaired super-short exposure image, a repaired normal exposure image and a repaired long exposure image.

[0099] In another embodiment of the application, in order to make the mixed exposure multi-frame image more complete, the motion region can be repaired, as shown in FIG. 4, and then the step S103 is replaced by the following steps S401-S407. Figure 4

[0100] In step S401, the super-short exposure image, the denoised normal exposure image and the denoised long exposure image are globally registered based on the short exposure image to obtain a registered super-short exposure image, a second registered normal exposure image and a second registered long exposure image.

[0101] As an optional implementation, the way in which the super-short exposure image, the denoised normal exposure image and the denoised long exposure image are globally registered based on the short exposure image to obtain a registered super-short exposure image, a second registered normal exposure image and a second registered long exposure image in step S401 can include the following steps:

[0102] In step S401, the super-short exposure image, the denoised normal exposure image and the denoised long exposure image are globally registered based on the short exposure image to obtain a registered super-short exposure image, a second registered normal exposure image and a second registered long exposure image.

[0103] In step S401, the super-short exposure image, the denoised normal exposure image and the denoised long exposure image are globally registered based on the short exposure image to obtain a registered super-short exposure image, a second registered normal exposure image and a second registered long exposure image. ​

[0104] determining a third homography matrix of the brightness-aligned ultra-short exposure image, the brightness-aligned normal exposure image and the brightness-aligned long exposure image according to ORB features of the short exposure image, the brightness-aligned ultra-short exposure image, the brightness-aligned normal exposure image and the brightness-aligned long exposure image;

[0105] performing global registration on the ultra-short exposure image, the denoised normal exposure image and the denoised long exposure image based on the third homography matrix and the short exposure image to obtain a registered ultra-short exposure image, a second registered normal exposure image and a second registered long exposure image.

[0106] In this embodiment, the brightness of the four types of images can be matched with each other by performing brightness alignment on the ultra-short exposure image, the denoised normal exposure image and the denoised long exposure image based on the brightness value of the short exposure image, so that the four types of images can be better fused.

[0107] In the embodiment, the exposure values of the ultra-short exposure image, the short exposure image, the denoised normal exposure image and the denoised long exposure image are different, so the images can be aligned in brightness value.

[0108] The brightness value Ev of the short exposure image is taken as a reference. -1 For reference:

[0109]

[0110] wherein sl represents the ultra-short exposure image, l represents the short exposure image, m represents the normal exposure image and h represents the long exposure image.

[0111] After brightness alignment, each image can be divided by an octree, and then ORB features thereof are calculated, denoted as orb sl , orb l , orb m , orb h The short exposure image is taken as a reference image, and a standard RANSAC algorithm feature is used for brightness alignment, and a third homography matrix between different images can be estimated.

[0112] Finally, a bicubic interpolation algorithm can be used to perform global registration on the ultra-short exposure image, the denoised normal exposure image and the denoised long exposure image to obtain a registered ultra-short exposure image, a second registered normal exposure image and a second registered long exposure image.

[0113] Step S402, obtaining a third sum of absolute differences of each pixel in the short exposure image according to the pixel value of each pixel in the short exposure image, the registered ultra-short exposure image, the second registered normal exposure image and the second registered long exposure image.

[0114] Step S403, determining a region represented by the pixel whose third sum of absolute differences is greater than or equal to the second preset threshold in the short exposure image as a third motion region.

[0115] In the embodiment of the present application, if the third sum of absolute differences is greater than or equal to the second preset threshold, it can be considered that the brightness value of the pixel is too high and needs to be repaired, so the region represented by the pixel can be determined as the third motion region, that is, the third motion region needs to be repaired.

[0116] Step S404, determining a region represented by the pixel whose third sum of absolute differences is less than the second preset threshold and greater than 0 in the short exposure image as a fourth motion region.

[0117] In the embodiment of the present application, if the third sum of absolute differences is less than the second preset threshold, it can be considered that the brightness of the pixel does not match the global brightness and needs to be aligned, so the region represented by the pixel can be determined as the fourth motion region, that is, the fourth motion region needs to be aligned.

[0118] Step S405, determining the region corresponding to the third motion region in the registered ultra-short exposure image, the second registered normal exposure image and the second registered long exposure image as the third motion region.

[0119] Step S406, determining the region corresponding to the fourth motion region in the registered ultra-short exposure image, the second registered normal exposure image and the second registered long exposure image as the fourth motion region.

[0120] Step S407, performing pixel value repair on the third motion region and ghost removal on the fourth motion region to obtain a repaired ultra-short exposure image, a repaired normal exposure image and a repaired long exposure image.

[0121] Implementing the above steps S401 to S407, there can be various local motion regions corresponding to local motion or motion regions of motion scenes in high light regions in the mixed exposure multi-frame image. The motion region corresponding to such a scene usually has no similar content to fill, so the motion region needs to be repaired to make the mixed exposure multi-frame image more complete.

[0122] In the embodiment of the present application, the pixel value repairing method can be implemented by using a priority calculation method. p The image block Ψ p may be centered at a point p, and the priority P(p) of the point p can be defined, and the calculation formula can be:

[0123] P(p)=C(p)D(p)

[0124] Wherein, C(p) is a confidence term, D(p) is a data term, and their definitions are as follows:

[0125]

[0126] Wherein, |Ψ p | represents the number of pixels in the image block Ψ p , and α is a normalization factor, n p is the normal vector on the boundary δΩ; the priority P(p) is calculated on all the image blocks of the boundary points, and for the confidence C(p), the initial value is 0 in the to-be-filled region Ω, and the initial value is 1 in the non-filled region of the image where the third motion region is located.

[0127] After the priority of all the boundary points is calculated, the point with the highest priority can be selected for filling, assuming that the point with the highest priority is p The corresponding small block region centered at p Then the block with the highest similarity to p can be searched in the non-filled region of the image where the third motion region is located, and the block with the highest similarity is used to fill the block:

[0128]

[0129] After filling, the filled point can be set in the non-filled region, and the priorities of all points are updated, the point with the highest priority is found, and the filling is continued until the filling is completed.

[0130] Step S104, weighting and fusing the repaired ultra-short exposure image, the short exposure image, the repaired normal exposure image and the repaired long exposure image to obtain a fused image.

[0131] In the embodiment of the present application, before weighting and fusing the repaired ultra-short exposure image, the short exposure image, the repaired normal exposure image and the repaired long exposure image, the repaired ultra-short exposure image, the short exposure image, the repaired normal exposure image and the repaired long exposure image can also be aligned in brightness.

[0132] In the embodiment of the present application, each frame of image can be divided into five equal parts according to brightness, and the coordinates are {d0, d1, d2, d3, d4, d5}. Taking an ultra-low exposure image as an example, the mean value μ of the image in the interval [d3, d5] can be counted l and the variance Then the weight ω of the region (μ l -3σ l , μ l +3σ l ) is calculated according to the standard normal distribution sl Similarly, the weight ω of the low exposure in the interval [d2, d4] is calculated l , the weight ω of the medium exposure in the interval [d1, d3] is calculated m , and the weight ω of the long exposure in the interval [d0, d2] is calculated h Therefore, for a pixel point (x, y), the linear weighted fusion can be performed according to the weight calculated by each image, and the calculation formula is:

[0133]

[0134] In step S105, the noise of the fused image is reduced according to the brightness of the fused image, and a high dynamic range image is obtained.

[0135] In another embodiment of the present application, in order to make the noise variance of the obtained high dynamic range image more stable, the inverse variance stabilization transformation of the fused image after noise reduction can be realized based on a pre-constructed Gaussian-Poisson distribution table, as shown in Figure 5 Therefore, the above step S105 is replaced by the following steps S501-S504:

[0136] In step S501, the Gaussian-Poisson distribution parameters corresponding to each pixel value of the fused image are obtained from the pre-constructed Gaussian-Poisson distribution table.

[0137] In the embodiment of the present application, the construction method of the Gaussian-Poisson distribution table can be as follows: for any exposure image, 500 segments are divided according to the image brightness. Assuming that the maximum pixel value is pix_max, the length of each segment is step=pix_max / 500. The variance of the pixel points in the interval [i*step-0.5*step, i*step+0.5*step] of each mean value is estimated, and the mean value-variance point pair of the exposure image is drawn According to the noise variance of the image, the data is divided into four segments, and the parameters (a i , b i ) of the Gaussian-Poisson distribution of the four segments are estimated by the least square method:

[0138]

[0139] Wherein, a is Poisson distribution parameter, b is Gaussian distribution parameter, through least square parameter estimation, we obtain four groups of parameters (a i ,b i ), i=1, 2, 3, 4, namely Gaussian-Poisson distribution table is obtained.

[0140] Step S502, according to the individual pixel value and the Gaussian-Poisson distribution parameter, the transformed pixel value is obtained.

[0141] In the embodiment of the application, for all pixel values of the fusion image, the Gaussian-Poisson distribution parameter (a i ,b i ) can be found according to the luminance segmentation in the Gaussian-Poisson distribution table, the variance is stably changed, and the transformed pixel value is obtained, and the change formula is:

[0142]

[0143] Wherein, s is the pixel value before transformation, t is the pixel value after transformation, μ i is the mean value of the calibrated Gaussian distribution, and the mean value of the Gaussian distribution is usually 0.

[0144] Step S503, according to the preset edge weight and the transformed pixel value, the fusion image is denoised to obtain a denoised fusion image.

[0145] In the embodiment of the application, the fusion image can be denoised by a weighted guided filter, so that the fusion image has better edge retention characteristics.

[0146] Specifically, the guide image G can be analyzed, and for the pixel point p', the variance in the 3x3 neighborhood Ω1(p') of p' is Therefore, for the whole image, the following edge index can be obtained:

[0147]

[0148] Wherein ε is a very small parameter, and is open to the outside world, in order to prevent the image from being linearly affected by the edge weight Γ G (p'), the edge weight Γ g (p') can be simply Gaussian filtered;

[0149] After weight calculation, the weight can be introduced into the guided filter by means of image weight distribution, and the weighted guided filter is re-derived:

[0150]

[0151] The optimal solution is obtained as follows:

[0152]

[0153] b p′ = μ X (p') - a p′ μ G (p')

[0154] wherein, is two matrix element point-to-point multiplication,

[0155] Different denoising intensities are adopted to filter each brightness area of the image by means of the weighted steering filter, so that the image after denoising is obtained, μ is the mean value, and the final filtering formula is:

[0156]

[0157] Here, and are the mean values of a and b in the 3x3 window; the fused image is denoised by the filtering formula, so that the fused image after denoising is obtained. p p

[0158] Step S504: performing inverse variance stabilization transformation on the fused image after denoising to obtain a high dynamic range image.

[0159] In the embodiment of the application, the inverse variance stabilization transformation can use optimal unbiased variance stabilization transformation, and for each pixel point, the transformation formula can be:

[0160]

[0161] The steps S501-S504 are implemented, and the inverse variance stabilization transformation of the fused image after denoising can be realized based on the pre-constructed Gaussian-Poisson distribution table, so that the noise variance of the obtained high dynamic range image is more stable.

[0162] As an optional implementation, the following steps can be further performed after step S105:

[0163] performing edge-preserving filtering on the high dynamic range image to obtain a basic structure layer of the high dynamic range image;

[0164] obtaining a detail layer of the high dynamic range image according to the high dynamic range image and the basic structure layer;

[0165] performing global tone mapping on the basic structure layer to obtain a basic structure image;

[0166] obtaining a global tone mapping image according to the basic structure image and the detail layer;

[0167] ​​​​Performing local tone mapping on the global tone mapping image to obtain a high dynamic range image with strong contrast.

[0168] In this embodiment, the high dynamic range image is first edge-preserving filtered, and the basic structure layer of the high dynamic range image is globally tone mapped to reduce the bit width of the high dynamic range image to the target bit width, and then the high dynamic range image is locally tone mapped to obtain a high dynamic range image with strong contrast.

[0169] In the embodiment, the data bit width of the high dynamic range image is high, and the current display usually uses an 8-bit image display system, so the high dynamic range image needs to be globally tone mapped to reduce the bit width to the bit width consistent with the display system. However, the image after global tone mapping is usually not enough in contrast, and needs to be locally tone mapped to enhance the local contrast of the image.

[0170] Specifically, the edge-preserving filtering method can be:

[0171] In order to preserve the details and contrast of the high dynamic range image and prevent the halo effect caused by tone mapping, the high dynamic range image is edge-preserving filtered to obtain a basic structure layer I base The edge-preserving filtering can be a side window filter, which has good edge-preserving characteristics and fast calculation speed while filtering, and the specific process is as follows:

[0172] (1) Select S={0°, 45°, 90°, 135°, 180°, 235°, 270°, 315°} eight directions;

[0173] (2) Select a window size in the selected eight directions, and perform box filtering in the window;

[0174] (3) Select the point with the smallest error from the eight filtering results as the filtering result, that is:

[0175]

[0176] where I n is the box filtering result in the n∈S direction. And the high dynamic range image can be subtracted from the basic structure layer to obtain a detail layer I detail of the high dynamic range image, that is:

[0177]

[0178] In the embodiment of the present application, the global tone mapping mode can be: using global tone mapping on the basic structure layer of the high dynamic range image to reduce the bit width of the image to the target bit width; the global tone mapping calculation formula can be:

[0179]

[0180] wherein, I is I base , D is the basic structure image after global tone mapping, D max is the maximum pixel value of the display system, D min is the minimum pixel value of the display system, and τ is obtained by adaptive calculation on the basic structure image, and is used to adaptively control the global tone mapping of the image, and the calculation method of τ is as follows:

[0181] (1) Calculate the log mean of I base

[0182]

[0183] wherein, N is the pixel number of the high dynamic range image, and ε is a very small number, which is used to prevent the calculation when the pixel value of the high dynamic range image is 0;

[0184] (2) Calculate the k value used to control the brightness of the image

[0185]

[0186] wherein, A and B are empirical values, which are respectively set as 0.4 and 2, so that the value range of k is [0.2, 0.8];

[0187] (3) Calculate τ, and the formula of τ is as follows

[0188]

[0189] The above equation can be solved by a numerical root method (such as Newton method), and τ can be obtained; and then the basic structure image I base-gtm after global tone mapping is obtained, and then the details of the image are added back to obtain the global tone mapping image, that is,

[0190] I gtm = I base-gtm + I detail

[0191] In the embodiment of the present application, the local tone mapping mode can be: using an improved local tone mapping algorithm based on the retinex theory, and the steps are as follows:

[0192] (1) Image global adaptive calculation:

[0193] ​

[0194] wherein, is the log mean, and its calculation process is consistent with the log mean calculation method in global tone mapping, which is not described here;

[0195] (2) Image local adaptive calculation mapping:

[0196] I l (x, y) = log(I g (x, y)) - log(H g (x, y))

[0197] wherein, H g is the result of side window filtering on I g , that is, the result of local tone mapping is:

[0198]

[0199] wherein, α(x, y) is a contrast control factor, and its calculation formula is:

[0200]

[0201] wherein, η is a control parameter, adjusting the contrast of the image; β is a nonlinearity control parameter, and its calculation formula is:

[0202]

[0203] After local tone mapping, a high dynamic range image with strong contrast can be obtained, which can be conveniently displayed by a display system, and the high dynamic range image with strong contrast has very good contrast.

[0204] The present application can realize noise reduction for different regions in the image according to the actual situation of the regions, avoid over-reduction or insufficient reduction in some regions, and improve the clarity of the image. In addition, the present application can ensure that the multiple images obtained by mixed exposure are more suitable for generating a high dynamic range image. In addition, the present application can make the scene classification model output an accurate exposure strategy according to the input environmental information. In addition, the present application can make the error control of the low-light region in the normally exposed image more strict. In addition, the present application can make the error control of the low-light region in the long-exposed image more strict. In addition, the present application can effectively improve the signal-to-noise ratio of the dark image signal of the normally exposed image and the long-exposed image. In addition, the present application can make the noise reduction effect for the normally exposed image and the long-exposed image better. In addition, the present application can better fuse the four types of images. In addition, the present application can make the multiple images obtained by mixed exposure more complete. In addition, the present application can make the noise variance of the obtained high dynamic range image more stable. In addition, the present application can obtain a high dynamic range image with strong contrast.

[0205] After introducing the method of the exemplary embodiment of the present application, next, with reference to Figure 6 A high dynamic range mixed exposure imaging device according to an exemplary embodiment of the present application is described, the device comprising:

[0206] The acquisition unit 601 is configured to acquire multiple images obtained by mixed exposure, wherein the multiple images obtained by mixed exposure at least include an ultra-short exposure image, a short exposure image, a normally exposed image, and a long exposure image.

[0207] The first noise reduction unit 602 is configured to respectively perform noise reduction on the normally exposed image and the long exposure image acquired by the acquisition unit 601, to obtain a normally exposed image after noise reduction and a long exposure image after noise reduction.

[0208] The repair unit 603 is configured to repair the ultra-short exposure image, the normally exposed image after noise reduction, and the long exposure image after noise reduction acquired by the acquisition unit 601 and obtained by the first noise reduction unit 602, to obtain a repaired ultra-short exposure image, a repaired normally exposed image, and a repaired long exposure image.

[0209] The fusion unit 604 is configured to perform weighted fusion on the repaired ultra-short exposure image, the short exposure image, the repaired normally exposed image, and the repaired long exposure image obtained by the repair unit 603, to obtain a fused image.

[0210] The second noise reduction unit 605 is configured to perform noise reduction on the fused image according to the brightness of the fused image obtained by the fusion unit 604, to obtain a high dynamic range image.

[0211] As an optional implementation, the multi-frame image of mixed exposure is acquired by an image acquisition device, and the acquisition unit 601 can acquire the multi-frame image of mixed exposure in the following manner:

[0212] acquiring a current environment in which the image acquisition device is located;

[0213] determining an exposure strategy matched with the current environment;

[0214] acquiring the multi-frame image of mixed exposure according to the exposure strategy.

[0215] In this implementation, because the light and the dynamic range are different in different environments, different exposure strategies need to be set for different environments, so that the multi-frame image of mixed exposure acquired is more suitable for generating a high dynamic range image.

[0216] As an optional implementation, the acquisition unit 601 is further configured to:

[0217] acquire a training data set, the training data set including at least environment information and a corresponding environment category;

[0218] train a scene classification model based on the training data set, to obtain a trained scene classification model;

[0219] input the current environment into the scene classification model, to obtain an exposure strategy matched with the current environment.

[0220] In this implementation, the scene classification model can be trained according to a large amount of training data set, so that the scene classification model can accurately output an accurate exposure strategy according to input environment information.

[0221] As an optional implementation, the number of the normal exposure images and the long exposure images is multi-frame, and the first noise reduction unit 602 can perform noise reduction on the normal exposure images and the long exposure images respectively, to obtain a normal exposure image after noise reduction and a long exposure image after noise reduction, in the following manner:

[0222] selecting a reference normal exposure image from the multi-frame normal exposure images;

[0223] selecting a reference long exposure image from the multi-frame long exposure images;

[0224] determine a first homography matrix of normal exposure images other than the reference normal exposure image in the plurality of normal exposure images according to the reference normal exposure image;

[0225] determine a second homography matrix of long exposure images other than the reference long exposure image in the plurality of long exposure images according to the reference long exposure image;

[0226] register the normal exposure images other than the reference normal exposure image based on the first homography matrix and the reference normal exposure image, to obtain first registered normal exposure images;

[0227] register the long exposure images other than the reference long exposure image based on the second homography matrix and the reference long exposure image, to obtain first registered long exposure images;

[0228] perform spatial domain noise reduction on the first motion region in the normal exposure images, the first motion region in the first registered normal exposure images, the second motion region in the long exposure images, and the second motion region in the first registered long exposure images, to obtain a noise-reduced normal exposure image and a noise-reduced long exposure image.

[0229] In this embodiment, since the normal exposure images and the long exposure images are usually overexposed in bright parts, the effective content is limited, and the effective content is mainly concentrated in dark parts. Therefore, the multi-frame superposition noise reduction of the normal exposure images and the long exposure images can effectively improve the signal-to-noise ratio of the dark part image signals.

[0230] As an optional embodiment, the manner in which the first noise reduction unit 602 determines the first homography matrix of the normal exposure images other than the reference normal exposure image in the plurality of normal exposure images according to the reference normal exposure image can be as follows:

[0231] determine the ORB features of the reference normal exposure image and a target normal exposure image other than the reference normal exposure image in the plurality of normal exposure images;

[0232] determine the first homography matrix of each target normal exposure image according to the ORB features of the reference normal exposure image and the ORB features of each target normal exposure image.

[0233] In the implementation of the embodiment, the homography matrix describes the global spatial position relationship between two images, including translation, rotation, scaling, and affine relationship. Since the pixel region used for high dynamic range imaging of the normal exposure image is mainly the medium and low brightness region, in the error formula for calculating the homography matrix, the pixel value error of the medium and low brightness region can be given a higher weight, so that the error control of the medium and low brightness region of the normal exposure image is more stringent.

[0234] As an optional implementation, the first noise reduction unit 602 determines the second homography matrix of the long exposure image other than the reference long exposure image in the plurality of long exposure images according to the reference long exposure image. The manner can be specifically as follows:

[0235] Determine the ORB features of the reference long exposure image and the target long exposure image other than the reference long exposure image in the plurality of long exposure images.

[0236] According to the ORB features of the reference long exposure image and the ORB features of the target long exposure image, the second homography matrix of each target long exposure image is determined.

[0237] In the implementation of the embodiment, the homography matrix describes the global spatial position relationship between two images, including translation, rotation, scaling, and affine relationship. Since the pixel region used for high dynamic range imaging of the normal exposure image is mainly the medium and low brightness region, in the error formula for calculating the homography matrix, the pixel value error of the medium and low brightness region can be given a higher weight, so that the error control of the medium and low brightness region of the normal exposure image is more stringent.

[0238] As an optional implementation, the first noise reduction unit 602 performs spatial domain noise reduction on the first motion region in the normal exposure image, the first motion region in the first registered normal exposure image, the second motion region in the long exposure image, and the second motion region in the first registered long exposure image, to obtain the noise-reduced normal exposure image and the noise-reduced long exposure image. The manner can be specifically as follows:

[0239] According to the pixel value of each pixel in the reference normal exposure image and the pixel value of each pixel in the first registered normal exposure image, the first sum of absolute differences of each pixel in the reference normal exposure image is obtained.

[0240] According to the pixel value of each pixel in the reference long exposure image and the pixel value of each pixel in the first registered long exposure image, the second sum of absolute differences of each pixel in the reference long exposure image is obtained.

[0241] determine a region represented by pixels with a first absolute sum of difference greater than or equal to a first preset threshold in the reference normal exposure image as a first motion region;

[0242] determine a region corresponding to the first motion region in the first registered normal exposure image as the first motion region as well;

[0243] determine a region represented by pixels with a second absolute sum of difference greater than or equal to the first preset threshold in the reference long exposure image as a second motion region;

[0244] determine a region corresponding to the second motion region in the first registered long exposure image as the second motion region as well;

[0245] perform spatial noise reduction on the first motion region and the second motion region to obtain a denoised normal exposure image and a denoised long exposure image.

[0246] In this embodiment, various local motions such as non-rigid deformation may exist in the normal exposure image and the long exposure image, so the motion regions of the local motions in the normal exposure image and the long exposure image can be determined first, and then noise reduction can be performed on the motion regions, so that the noise reduction effect on the normal exposure image and the long exposure image is better.

[0247] As an optional embodiment, the repairing unit 603 repairs the ultra-short exposure image, the denoised normal exposure image, and the denoised long exposure image to obtain a repaired ultra-short exposure image, a repaired normal exposure image, and a repaired long exposure image in the following manner:

[0248] perform global registration on the ultra-short exposure image, the denoised normal exposure image, and the denoised long exposure image based on the short exposure image to obtain a registered ultra-short exposure image, a second registered normal exposure image, and a second registered long exposure image;

[0249] obtain a third absolute sum of difference of each pixel in the short exposure image according to the pixel values of each pixel in the short exposure image, the registered ultra-short exposure image, the second registered normal exposure image, and the second registered long exposure image;

[0250] determine a region represented by pixels with a third absolute sum of difference greater than or equal to a second preset threshold in the short exposure image as a third motion region;

[0251] determine a region represented by pixels with a third absolute sum of difference less than the second preset threshold and greater than 0 in the short exposure image as a fourth motion region;

[0252] The region corresponding to the third motion region in the registered ultra-short exposure image, the second registered normal exposure image and the second registered long exposure image is also determined as a third motion region;

[0253] The region corresponding to the fourth motion region in the registered ultra-short exposure image, the second registered normal exposure image and the second registered long exposure image is also determined as a fourth motion region;

[0254] The third motion region is pixel value patched, and the fourth motion region is ghosting removed, to obtain a patched ultra-short exposure image, a patched normal exposure image and a patched long exposure image.

[0255] In this embodiment, there can be various local motion regions corresponding to local motions or motion regions of motion scenes in highlight regions in the mixed exposure multi-frame image. The motion region corresponding to such a scene usually has no similar content to fill, and thus the motion region needs to be patched to make the mixed exposure multi-frame image more complete.

[0256] As an optional embodiment, the patching unit 603 performs global registration on the ultra-short exposure image, the denoised normal exposure image and the denoised long exposure image based on the short exposure image, to obtain a registered ultra-short exposure image, a second registered normal exposure image and a second registered long exposure image. The manner can be specifically as follows:

[0257] Based on the luminance value of the short exposure image, the ultra-short exposure image, the denoised normal exposure image and the denoised long exposure image are subjected to luminance alignment, to obtain a luminance-aligned ultra-short exposure image, a luminance-aligned normal exposure image and a luminance-aligned long exposure image;

[0258] The ORB features of the short exposure image, the luminance-aligned ultra-short exposure image, the luminance-aligned normal exposure image and the luminance-aligned long exposure image are determined;

[0259] According to the ORB features of the short exposure image, the luminance-aligned ultra-short exposure image, the luminance-aligned normal exposure image and the luminance-aligned long exposure image, a third homography matrix of the luminance-aligned ultra-short exposure image, the luminance-aligned normal exposure image and the luminance-aligned long exposure image is determined;

[0260] Perform global registration on the ultra-short exposure image, the normal exposure image after noise reduction and the long exposure image after noise reduction based on the third homography matrix and the short exposure image, to obtain a registered ultra-short exposure image, a second registered normal exposure image and a second registered long exposure image.

[0261] In this way, the brightness of the four types of images can be matched with each other, so that the four types of images can be fused better.

[0262] As an optional implementation, the second noise reduction unit 605 can perform noise reduction on the fused image according to the brightness of the fused image, and the manner of obtaining the high dynamic range image can be as follows:

[0263] Obtain the Gaussian-Poisson distribution parameters corresponding to each pixel value of the fused image from a pre-constructed Gaussian-Poisson distribution table;

[0264] Obtain the transformed pixel value according to the each pixel value and the Gaussian-Poisson distribution parameter;

[0265] Perform noise reduction on the fused image according to the preset edge weight and the transformed pixel value, to obtain a noise-reduced fused image;

[0266] Perform inverse variance stabilization transformation on the noise-reduced fused image, to obtain a high dynamic range image.

[0267] In this way, the inverse variance stabilization transformation of the noise-reduced fused image can be implemented based on the pre-constructed Gaussian-Poisson distribution table, so that the noise variance of the obtained high dynamic range image is more stable.

[0268] As an optional implementation, the second noise reduction unit 605 can further be configured to:

[0269] After performing noise reduction on the fused image according to the brightness of the fused image to obtain a high dynamic range image, perform edge-preserving filtering on the high dynamic range image, to obtain a basic structure layer of the high dynamic range image;

[0270] Obtain a detail layer of the high dynamic range image according to the high dynamic range image and the basic structure layer;

[0271] Perform global tone mapping on the basic structure layer, to obtain a basic structure image;

[0272] Obtain a global tone mapping image according to the basic structure image and the detail layer;

[0273] performing local tone mapping on the global tone mapped image to obtain a high dynamic range image with strong contrast.

[0274] In this embodiment, the high dynamic range image is first edge-preserving filtered, and the basic structure layer of the high dynamic range image is globally tone mapped to reduce the bit width of the high dynamic range image to a target bit width, and then the high dynamic range image is locally tone mapped to obtain a high dynamic range image with strong contrast.

[0275] Having described the method and apparatus of the exemplary embodiments of the present application, next, reference is made to Figure 7 The computer readable storage medium of the exemplary embodiments of the present application is described as follows Figure 7 The computer readable storage medium shown in the figure is an optical disc 70, and a computer program (i.e. program product) is stored on the optical disc 70, which, when executed by a processor, implements each step described in the above method embodiments, for example, obtaining a plurality of mixed exposure images; wherein the plurality of mixed exposure images at least include an ultra-short exposure image, a short exposure image, a normal exposure image and a long exposure image; performing noise reduction on the normal exposure image and the long exposure image respectively to obtain a noise-reduced normal exposure image and a noise-reduced long exposure image; performing repair on the ultra-short exposure image, the noise-reduced normal exposure image and the noise-reduced long exposure image to obtain a repaired ultra-short exposure image, a repaired normal exposure image and a repaired long exposure image; performing weighted fusion on the repaired ultra-short exposure image, the short exposure image, the repaired normal exposure image and the repaired long exposure image to obtain a fused image; performing noise reduction on the fused image according to the brightness of the fused image to obtain a high dynamic range image; and the specific implementation of each step is not repeated here.

[0276] It should be noted that examples of the computer readable storage medium can also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical, magnetic storage medium, which will not be repeated here.

[0277] Having described the method, medium and apparatus of the exemplary embodiments of the present application, next, reference is made to Figure 8 The computing device for mixed exposure imaging for high dynamic range of the exemplary embodiments of the present application.

[0278] Figure 8A block diagram is shown of an exemplary computing device 80 suitable for implementing embodiments of the present invention, which may be a computer system or a server. Figure 8 The computing device 80 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0279] like Figure 8 As shown, the components of the computing device 80 may include, but are not limited to: one or more processors or processing units 801, system memory 802, and bus 803 connecting different system components (including system memory 802 and processing unit 801).

[0280] The computing device 80 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computing device 80, including volatile and non-volatile media, removable and non-removable media.

[0281] System memory 802 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 8021 and / or cache memory 8022. Computing device 80 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, ROM 8023 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 8 (Not shown in the image, usually referred to as "hard drive"). Although not shown in Figure 8 The diagram illustrates that disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disc drives for reading and writing to removable non-volatile optical discs (e.g., CD-ROMs, DVD-ROMs, or other optical media) can be provided. In these cases, each drive can be connected to bus 803 via one or more data media interfaces. System memory 802 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0282] A program / utility 8025 having a set (at least one) of program modules 8024 may be stored, for example, in system memory 802, and such program modules 8024 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment. Program modules 8024 typically perform the functions and / or methods described in the embodiments of the present invention.

[0283] The computing device 80 can also communicate with one or more external devices 804 such as a keyboard, a pointing device, a display, etc. through an input / output (I / O) interface 805. Further, the computing device 80 can communicate with one or more networks such as a local area network (LAN), a wide area network (WAN), and / or the public network, e.g., the Internet, through a network adapter 806. As depicted, the network adapter 806 is in communication with the other modules of the computing device 80, such as the processing unit 801, etc., through the bus 803. It should be appreciated that the computing device 80 can be connected to other types of computing devices, such as a network appliance, a server, a client, a personal computer, a programmable logic controller, a smart phone, a tablet computer, a network router, a network switch, a network bridge, etc. Figure 8 It should be appreciated that other hardware and / or software modules can be used in connection with the computing device 80, although not depicted in the ​

[0284] The processing unit 801 performs various function applications and data processing by running programs stored in the system memory 802, such as obtaining a multi-frame image of hybrid exposure, wherein the multi-frame image of hybrid exposure comprises at least an ultra-short exposure image, a short exposure image, a normal exposure image, and a long exposure image; performing noise reduction on the normal exposure image and the long exposure image respectively to obtain a noise-reduced normal exposure image and a noise-reduced long exposure image; performing repair on the ultra-short exposure image, the noise-reduced normal exposure image, and the noise-reduced long exposure image to obtain a repaired ultra-short exposure image, a repaired normal exposure image, and a repaired long exposure image; performing weighted fusion on the repaired ultra-short exposure image, the short exposure image, the repaired normal exposure image, and the repaired long exposure image to obtain a fused image; and performing noise reduction on the fused image according to the brightness of the fused image to obtain a high dynamic range image. The specific implementation of each step is not repeated here. It should be noted that although several units / modules or sub-units / sub-modules of the hybrid exposure imaging device of high dynamic range are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules.

[0285] In the description of the present application, it should be noted that the terms "first", "second", "third" are only for the purpose of description and cannot be understood as indicating or implying relative importance.

[0286] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0287] ​In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. The described device embodiments are merely schematic, and for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the among different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electric, mechanical or other forms.

[0288] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. In actual implementation, some or all of the units can be selected according to the actual needs to achieve the purposes of the embodiments of the present application.

[0289] In addition, each function unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit.

[0290] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various media that can store program codes.

[0291] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present application, and are used to illustrate the technical solutions of the present application, but are not intended to limit the present application. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0292] In addition, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in the particular order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, some steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps.

[0293] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the product embodiment described later, since it is corresponding to the method, the description is relatively simple, and the relevant part can be referred to the part of the system embodiment.

[0294] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, and the changes or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A hybrid exposure imaging method of high dynamic range, characterized in that, The method comprises: acquiring a plurality of mixed exposure images; wherein the plurality of mixed exposure images at least include an ultra-short exposure image, a short exposure image, a normal exposure image, and a long exposure image; respectively performing noise reduction on the normal exposure image and the long exposure image to obtain a noise-reduced normal exposure image and a noise-reduced long exposure image; performing global registration on the ultra-short exposure image, the noise-reduced normal exposure image, and the noise-reduced long exposure image based on the short exposure image to obtain a registered ultra-short exposure image, a second registered normal exposure image, and a second registered long exposure image; obtaining a third sum of absolute differences of each pixel in the short exposure image according to pixel values of each pixel in the short exposure image, the registered ultra-short exposure image, the second registered normal exposure image, and the second registered long exposure image; determining a region represented by a pixel with a third sum of absolute differences greater than or equal to a second preset threshold in the short exposure image as a third motion region; determining a region represented by a pixel with a third sum of absolute differences less than the second preset threshold and greater than 0 in the short exposure image as a fourth motion region; determining a region corresponding to the third motion region in the registered ultra-short exposure image, the second registered normal exposure image, and the second registered long exposure image as the third motion region; determining a region corresponding to the fourth motion region in the registered ultra-short exposure image, the second registered normal exposure image, and the second registered long exposure image as the fourth motion region; performing pixel value repair on the third motion region and ghosting elimination on the fourth motion region to obtain a repaired ultra-short exposure image, a repaired normal exposure image, and a repaired long exposure image; performing weighted fusion on the repaired ultra-short exposure image, the short exposure image, the repaired normal exposure image, and the repaired long exposure image to obtain a fused image; performing noise reduction on the fused image according to brightness of the fused image to obtain a high dynamic range image.

2. The hybrid exposure imaging method of high dynamic range according to claim 1, characterized in that, The plurality of mixed exposure images are acquired by an image acquisition device, and the acquiring of the plurality of mixed exposure images comprises: acquiring a current environment in which the image acquisition device is located; determining an exposure strategy matched with the current environment; acquiring a plurality of mixed exposure images according to the exposure strategy.

3. The hybrid exposure imaging method of high dynamic range according to claim 2, characterized in that, Before the determining of the exposure strategy matched with the current environment, the method further comprises: acquiring a training data set, wherein the training data set at least includes environment information and a corresponding environment category; training a scene classification model based on the training data set to obtain a trained scene classification model; the determining of the exposure strategy matched with the current environment comprises: inputting the current environment into the scene classification model to obtain the exposure strategy matched with the current environment.

4. The hybrid exposure imaging method of high dynamic range according to claim 1, wherein, The number of the normal exposure images and the long exposure images is multiple frames, the normal exposure images and the long exposure images are respectively subjected to noise reduction to obtain a normal exposure image after noise reduction and a long exposure image after noise reduction, and the method comprises the following steps: selecting a reference normal exposure image from the multiple normal exposure images; selecting a reference long exposure image from the multiple long exposure images; determining a first homography matrix of a normal exposure image other than the reference normal exposure image from the multiple normal exposure images according to the reference normal exposure image; determining a second homography matrix of a long exposure image other than the reference long exposure image from the multiple long exposure images according to the reference long exposure image; performing registration on the normal exposure image other than the reference normal exposure image based on the first homography matrix and the reference normal exposure image to obtain a first registered normal exposure image; performing registration on the long exposure image other than the reference long exposure image based on the second homography matrix and the reference long exposure image to obtain a first registered long exposure image; performing spatial domain noise reduction on a first motion region in the normal exposure image, a first motion region in the first registered normal exposure image, a second motion region in the long exposure image and a second motion region in the first registered long exposure image to obtain the normal exposure image after noise reduction and the long exposure image after noise reduction.

5. The hybrid exposure imaging method of high dynamic range according to claim 4, characterized in that, The method of determining the first homography matrix of the normal exposure image other than the reference normal exposure image from the multiple normal exposure images according to the reference normal exposure image comprises the following steps: determining ORB features of the reference normal exposure image and a target normal exposure image other than the reference normal exposure image from the multiple normal exposure images; determining the first homography matrix of each target normal exposure image according to the ORB features of the reference normal exposure image and the ORB features of each target normal exposure image.

6. The hybrid exposure imaging method of high dynamic range according to claim 5, wherein, The method of determining the second homography matrix of the long exposure image other than the reference long exposure image from the multiple long exposure images according to the reference long exposure image comprises the following steps: determining ORB features of the reference long exposure image and a target long exposure image other than the reference long exposure image from the multiple long exposure images; determining the second homography matrix of each target long exposure image according to the ORB features of the reference long exposure image and the ORB features of the target long exposure image.

7. The hybrid exposure imaging method of high dynamic range according to any one of claims 4 to 6, characterized in that, The method of performing spatial domain noise reduction on the first motion region in the normal exposure image, the first motion region in the first registered normal exposure image, the second motion region in the long exposure image and the second motion region in the first registered long exposure image to obtain the normal exposure image after noise reduction and the long exposure image after noise reduction comprises the following steps: According to the pixel value of each pixel in the reference normal exposure image and the pixel value of each pixel in the first registered normal exposure image, a first sum of absolute differences of each pixel in the reference normal exposure image is obtained; According to the pixel value of each pixel in the reference long exposure image and the pixel value of each pixel in the first registered long exposure image, a second sum of absolute differences of each pixel in the reference long exposure image is obtained; A region represented by a pixel whose first sum of absolute differences in the reference normal exposure image is greater than or equal to a first preset threshold is determined as a first motion region; A region corresponding to the first motion region in the first registered normal exposure image is also determined as the first motion region; A region represented by a pixel whose second sum of absolute differences in the reference long exposure image is greater than or equal to a first preset threshold is determined as a second motion region; A region corresponding to the second motion region in the first registered long exposure image is also determined as the second motion region; The first motion region and the second motion region are subjected to spatial domain noise reduction to obtain a denoised normal exposure image and a denoised long exposure image.

8. The hybrid exposure imaging method of high dynamic range according to claim 1, wherein, The global registration of the short exposure image, the denoised normal exposure image and the denoised long exposure image based on the short exposure image to obtain the registered ultra-short exposure image, the second registered normal exposure image and the second registered long exposure image, comprises: The brightness alignment of the ultra-short exposure image, the denoised normal exposure image and the denoised long exposure image based on the brightness value of the short exposure image to obtain the brightness-aligned ultra-short exposure image, the brightness-aligned normal exposure image and the brightness-aligned long exposure image; The ORB features of the short exposure image, the brightness-aligned ultra-short exposure image, the brightness-aligned normal exposure image and the brightness-aligned long exposure image are determined; The third homography matrix of the brightness-aligned ultra-short exposure image, the brightness-aligned normal exposure image and the brightness-aligned long exposure image is determined according to the ORB features of the short exposure image, the brightness-aligned ultra-short exposure image, the brightness-aligned normal exposure image and the brightness-aligned long exposure image; The global registration of the ultra-short exposure image, the denoised normal exposure image and the denoised long exposure image based on the third homography matrix and the short exposure image to obtain the registered ultra-short exposure image, the second registered normal exposure image and the second registered long exposure image.

9. The hybrid exposure imaging method of high dynamic range according to claim 1, wherein, The denoising of the fusion image according to the brightness of the fusion image to obtain a high dynamic range image, comprises: The Gaussian-Poisson distribution parameters corresponding to each pixel value of the fusion image are obtained from a pre-constructed Gaussian-Poisson distribution table; The transformed pixel value is obtained according to the each pixel value and the Gaussian-Poisson distribution parameter; The denoising of the fusion image according to the preset edge weight and the transformed pixel value to obtain a denoised fusion image; Perform inverse variance stabilization transformation on the noise-reduced fusion image to obtain a high dynamic range image.

10. The hybrid exposure imaging method of high dynamic range according to claim 1, wherein, After the high dynamic range image is obtained by performing noise reduction on the fusion image according to the brightness of the fusion image, the method further comprises: Performing edge-preserving filtering on the high dynamic range image to obtain a basic structure layer of the high dynamic range image; Obtaining a detail layer of the high dynamic range image according to the high dynamic range image and the basic structure layer; Performing global tone mapping on the basic structure layer to obtain a basic structure image; Obtaining a global tone mapping image according to the basic structure image and the detail layer; Performing local tone mapping on the global tone mapping image to obtain a high dynamic range image with strong contrast.

11. A hybrid exposure imaging apparatus of high dynamic range, characterized by, The device comprises: An acquisition unit configured to acquire a plurality of mixed exposure images; wherein the plurality of mixed exposure images at least include an ultra-short exposure image, a short exposure image, a normal exposure image, and a long exposure image; A first noise reduction unit configured to perform noise reduction on the normal exposure image and the long exposure image respectively to obtain a noise-reduced normal exposure image and a noise-reduced long exposure image; A repairing unit configured to perform global registration on the ultra-short exposure image, the noise-reduced normal exposure image, and the noise-reduced long exposure image based on the short exposure image to obtain a registered ultra-short exposure image, a second registered normal exposure image, and a second registered long exposure image; Obtaining a third sum of absolute differences of each pixel in the short exposure image according to the pixel values of each pixel in the short exposure image, the registered ultra-short exposure image, the second registered normal exposure image, and the second registered long exposure image; Determining a region represented by a pixel with a third sum of absolute differences greater than or equal to a second preset threshold in the short exposure image as a third motion region; Determining a region represented by a pixel with a third sum of absolute differences less than the second preset threshold and greater than 0 in the short exposure image as a fourth motion region; Determining a region corresponding to the third motion region in the registered ultra-short exposure image, the second registered normal exposure image, and the second registered long exposure image as the third motion region; Determining a region corresponding to the fourth motion region in the registered ultra-short exposure image, the second registered normal exposure image, and the second registered long exposure image as the fourth motion region; Performing pixel value repair on the third motion region and ghosting elimination on the fourth motion region to obtain a repaired ultra-short exposure image, a repaired normal exposure image, and a repaired long exposure image; A fusion unit configured to perform weighted fusion on the repaired ultra-short exposure image, the short exposure image, the repaired normal exposure image, and the repaired long exposure image to obtain a fusion image; A second noise reduction unit configured to perform noise reduction on the fusion image according to the brightness of the fusion image to obtain a high dynamic range image.

12. A computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to carry out the method of any one of claims 1 to 10.

13. A computing device, the computing device comprising: at least one processor, a memory, and an input output unit; wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program stored in the memory to perform the method of any one of claims 1 to 10.

Images