Methods, apparatuses, storage media and electronic devices for generating high dynamic range images

Abstract

This disclosure provides a method, apparatus, storage medium, and electronic device for generating high dynamic range (HDR) images. The method includes: acquiring a long-exposure visible light image, a long-exposure near-infrared image, a short-exposure visible light image, and a short-exposure near-infrared image of a target scene, wherein the short-exposure near-infrared image and the long-exposure near-infrared image are not overexposed, and the bright areas in the short-exposure visible light image are not overexposed; reconstructing the brightness of the overexposed areas in the long-exposure visible light image based on the long-exposure near-infrared image, the short-exposure visible light image, and the short-exposure near-infrared image; reconstructing the chromaticity of the overexposed areas in the long-exposure visible light image based on the short-exposure visible light image; and generating a HDR image based on the long-exposure visible light image and the brightness and chromaticity reconstruction results. The HDR image generated by this method can clearly image both bright and dark areas without artifacts.

Description

Technical Field

[0001] This disclosure relates to an image generation method, and more particularly to a method, apparatus, storage medium, and electronic device for generating high dynamic range images. Background Technology

[0002] High Dynamic Range (HDR) image fusion technology is a computer-generated scene image fusion technique that produces high-quality, realistic effects. It is widely used in computer game development, medical imaging, remote sensing image processing, and computer graphics. Dynamic range refers to the ratio of the maximum to the minimum brightness value of an image. A larger dynamic range indicates more scene detail and a more realistic visual effect. Traditional images typically use one byte (8 bits) to store a single pixel, resulting in only 256 brightness levels, which is insufficient for many applications requiring high scene detail. HDR images, however, use multiple bytes of floating-point numbers to store the brightness value of a single pixel, comprehensively representing the high dynamic range of natural scenes. Summary of the Invention

[0003] This disclosure provides a method, apparatus, storage medium, and electronic device for generating high dynamic range images, which are capable of generating clear high dynamic range images.

[0004] A first aspect of this disclosure provides a method for generating a high dynamic range (HDR) image. The method includes: acquiring a long-exposure visible light image, a long-exposure near-infrared image, a short-exposure visible light image, and a short-exposure near-infrared image of a target scene, wherein the short-exposure near-infrared image and the long-exposure near-infrared image are not overexposed, bright areas in the short-exposure visible light image are not overexposed, the spatial relationships of the long-exposure visible light image and the long-exposure near-infrared image are consistent, and the spatial relationships of the short-exposure visible light image and the short-exposure near-infrared image are consistent; reconstructing the brightness of overexposed areas in the long-exposure visible light image based on the long-exposure near-infrared image, the short-exposure visible light image, and the short-exposure near-infrared image; reconstructing the chromaticity of overexposed areas in the long-exposure visible light image based on the short-exposure visible light image; and generating a HDR image based on the long-exposure visible light image and its brightness and chromaticity reconstruction results.

[0005] In one embodiment of the first aspect, reconstructing the brightness of the overexposed area of ​​the long-exposure visible light image based on the long-exposure near-infrared image, the short-exposure visible light image, and the short-exposure near-infrared image includes: obtaining a mapping relationship between the brightness of the short-exposure visible light image and the near-infrared intensity of the short-exposure near-infrared image as a first mapping relationship; obtaining a spatial matching relationship between the long-exposure near-infrared image and the short-exposure near-infrared image; obtaining an image block correspondence relationship between the long-exposure visible light image and the short-exposure visible light image as a second mapping relationship based on the spatial matching relationship between the long-exposure near-infrared image and the short-exposure near-infrared image; obtaining a mapping relationship between the brightness of the long-exposure visible light image and the near-infrared intensity of the long-exposure near-infrared image as a third mapping relationship based on the first mapping relationship and the second mapping relationship; and reconstructing the brightness of the overexposed area of ​​the long-exposure visible light image based on the third mapping relationship and the near-infrared intensity of the long-exposure near-infrared image.

[0006] In one embodiment of the first aspect, reconstructing the brightness of the overexposed region of the long-exposure visible light image includes: mapping the gradient of the overexposed region corresponding to the long-exposure near-infrared image to the gradient of the overexposed region of the long-exposure visible light image according to the third mapping relationship, wherein the overexposed region corresponds to the overexposed region of the long-exposure visible light image; obtaining the gradient of the long-exposure visible light image based on the gradient of the overexposed region and the gradient of the non-overexposed region; and reconstructing the brightness of the overexposed region of the long-exposure visible light image based on the gradient of the long-exposure visible light image.

[0007] In one embodiment of the first aspect, obtaining the mapping relationship between the brightness of the short-exposure visible light image and the near-infrared intensity of the short-exposure near-infrared image as a first mapping relationship includes: dividing the short-exposure scene into several regions based on the short-exposure visible light image and the short-exposure near-infrared image; and obtaining the first mapping relationship based on the brightness and near-infrared intensity of the visible light image in each region.

[0008] In one embodiment of the first aspect, for a region, the first mapping relationship corresponding to the region is as follows: V = a × N + b, where V represents the visible light brightness of the pixel in the region, N represents the near-infrared intensity of the pixel in the region, and a and b are mapping parameters, which depend on the visible light brightness and near-infrared intensity of the pixel in the region.

[0009] In one embodiment of the first aspect, for a region, the first mapping relationship corresponding to the region is as follows: V = a × N, where V represents the visible light brightness of the pixel in the region, N represents the near-infrared intensity of the pixel in the region, and a is a mapping parameter that depends on the visible light brightness and near-infrared intensity of the pixel in the region.

[0010] In one embodiment of the first aspect, segmenting a short-exposure scene into several regions includes: segmenting each channel of the short-exposure visible light image to obtain a first segmentation result; segmenting the short-exposure near-infrared image to obtain a second segmentation result; and obtaining a segmentation result of the short-exposure scene based on the first segmentation result and the second segmentation result.

[0011] In one embodiment of the first aspect, segmenting each channel of the short-exposure visible light image includes: obtaining a multi-level grayscale histogram of a channel; and obtaining the grayscale peak value of the channel in the multi-level grayscale histogram, and segmenting the pixels of the channel using the valley between adjacent grayscale peak values ​​as the segmentation boundary.

[0012] In one embodiment of the first aspect, performing chromaticity reconstruction of the overexposed area of ​​the long-exposure visible light image based on the short-exposure visible light image includes: obtaining a corresponding patch of the overexposed area of ​​the long-exposure visible light image in the short-exposure visible light image; and performing chromaticity reconstruction of the overexposed area of ​​the long-exposure visible light image based on the chromaticity of the corresponding patch.

[0013] In one embodiment of the first aspect, acquiring the long-exposure visible light image, the long-exposure near-infrared image, the short-exposure visible light image, and the short-exposure near-infrared image includes: acquiring a long-exposure raw image file and a short-exposure raw image file, wherein the long-exposure raw image file and the short-exposure raw image file are acquired by an image acquisition device using different exposure parameters on a target scene; and performing de-mosaic processing on the long-exposure raw image file to obtain the long-exposure visible light image and the long-exposure near-infrared image, and performing de-mosaic processing on the short-exposure raw image file to obtain the short-exposure visible light image and the short-exposure near-infrared image.

[0014] In one embodiment of the first aspect, the method for generating a high dynamic range image further includes: adjusting the exposure parameters of the image acquisition device so that neither the short-exposure visible light image nor the long-exposure near-infrared image is overexposed, and ensuring that the bright areas of the short-exposure visible light image are not overexposed.

[0015] In one embodiment of the first aspect, neither the short-exposure near-infrared image nor the long-exposure near-infrared image is overexposed, so that information of most areas in the short-exposure near-infrared image and the long-exposure near-infrared image can be captured, and that for overexposed areas in the long-exposure visible light image, near-infrared image information is retained in the corresponding areas of the short-exposure near-infrared image and the long-exposure near-infrared image; the bright areas in the short-exposure visible light image are not overexposed, so that the brightness information and chromaticity information of the bright areas are retained.

[0016] A second aspect of this disclosure provides an apparatus for generating a high dynamic range image. The apparatus includes: an image acquisition module configured to acquire a long-exposure visible light image, a long-exposure near-infrared image, a short-exposure visible light image, and a short-exposure near-infrared image of a target scene, wherein the short-exposure near-infrared image and the long-exposure near-infrared image are not overexposed, the bright areas of the short-exposure visible light image are not overexposed, the spatial relationships of the long-exposure visible light image and the long-exposure near-infrared image are consistent, and the spatial relationships of the short-exposure visible light image and the short-exposure near-infrared image are consistent; a brightness reconstruction module configured to reconstruct the brightness of the overexposed areas of the long-exposure visible light image based on the long-exposure near-infrared image, the short-exposure visible light image, and the short-exposure near-infrared image; a chromaticity reconstruction module configured to reconstruct the chromaticity of the overexposed areas of the long-exposure visible light image based on the short-exposure visible light image; and an image reconstruction module configured to generate a high dynamic range image based on the long-exposure visible light image and its brightness and chromaticity reconstruction results.

[0017] A third aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon. The computer program is executed to implement the method according to any one of the first aspects of this disclosure.

[0018] A fourth aspect of this disclosure provides an electronic device. The electronic device includes: a memory configured to store a computer program; and a processor configured to invoke the computer program to perform the method according to any one of the first aspects of this disclosure.

[0019] The method for generating a high dynamic range image according to embodiments of this disclosure involves reconstructing the brightness and color of overexposed areas in a long-exposure visible light image, and generating a high dynamic range image based on the reconstruction results and the long-exposure visible light image. The resulting high dynamic range image exhibits clear imaging of both bright and dark areas without artifacts. Attached Figure Description

[0020] Figure 1 The flowchart shown is a method for generating high dynamic range images in an embodiment of this disclosure.

[0021] Figure 2The flowchart shown is a process for brightness reconstruction in an embodiment of this disclosure.

[0022] Figure 3 The diagram shown is a schematic representation of the brightness reconstruction process in an embodiment of this disclosure.

[0023] Figure 4 The flowchart shown is a process for brightness reconstruction in an embodiment of this disclosure.

[0024] Figure 5A The flowchart shown is a process for obtaining the first mapping relationship in an embodiment of this disclosure.

[0025] Figure 5B The image shown is an example of a short-exposure visible light image from an embodiment of this disclosure.

[0026] Figure 5C The image shown is an example of a short-exposure near-infrared image from an embodiment of this disclosure.

[0027] Figure 5D The image shown is an example of the short exposure scene segmentation results in an embodiment of this disclosure.

[0028] Figure 6A The flowchart shown is a process for obtaining short-exposure scene segmentation results in an embodiment of this disclosure.

[0029] Figure 6B The flowchart shown is a process for segmenting channels in an embodiment of this disclosure.

[0030] Figure 7 The flowchart shown is a process for chromaticity reconstruction in an embodiment of this disclosure.

[0031] Figure 8A The flowchart shown is a process for acquiring short-exposure visible light images and short-exposure near-infrared images in embodiments of this disclosure.

[0032] Figure 8B and Figure 8C The following are examples of array layout diagrams of image acquisition devices in embodiments of this disclosure.

[0033] Figure 9 The diagram shown is a structural schematic of an apparatus for generating high dynamic range images in an embodiment of this disclosure.

[0034] Figure 10 The diagram shown is a schematic representation of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0035] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0036] It should be noted that the illustrations provided in the following embodiments are merely schematic representations of the basic concept of this disclosure. The illustrations only show components relevant to this disclosure and are not drawn according to the actual number, shape, and size of the components in implementation. In actual implementation, the form, quantity, and proportion of each component can be arbitrarily changed, and the component layout may also be more complex. Furthermore, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0037] High dynamic range imaging technology captures images at multiple exposure times in succession and fuses the image information to ultimately present a dynamic range compressed result, which can simultaneously and clearly display scene details in both dark and bright areas.

[0038] However, current HDR algorithms typically assume that multi-exposure images are perfectly aligned. In real-world image or video shooting scenarios, due to the instability of handheld cameras, the presence of moving objects, and other issues such as image misalignment, image shake, and object occlusion, the use of existing HDR algorithms often results in artifacts in the fused images, failing to meet actual needs.

[0039] This disclosure addresses the aforementioned issues such as misalignment between long and short exposure images, image jitter, and object occlusion by proposing a method for generating high dynamic range (HDR) images through image HDR fusion enhancement. The HDR images generated by this method can clearly capture both bright and dark parts without artifacts.

[0040] The following will describe specific implementations of the method for generating high dynamic range images provided in this disclosure with reference to the accompanying drawings through exemplary embodiments.

[0041] Figure 1This is a flowchart illustrating a method for generating a high dynamic range image according to an embodiment of the present disclosure. Each step in this flowchart can be implemented by one or more specific modules. In some embodiments, these modules can be modules in chips such as GPUs (graphics processing units), DSPs (digital signal processing units), etc. Figure 1 As shown, the method for generating a high dynamic range image in this embodiment includes the following steps S11 to S14.

[0042] In step S11, long-exposure visible light images, long-exposure near-infrared images, short-exposure visible light images, and short-exposure near-infrared images of the target scene are acquired. The short-exposure near-infrared and long-exposure near-infrared images are not overexposed, and bright areas in the short-exposure visible light images are not overexposed. The spatial relationships of the long-exposure visible light and long-exposure near-infrared images are consistent. The exposure time of the long-exposure visible light and long-exposure near-infrared images is longer than that of the short-exposure visible light and short-exposure near-infrared images. Due to the longer exposure time of the long-exposure visible light images, the clarity of dark areas in the long-exposure visible light images is relatively higher. The range of bright and dark areas can be configured according to actual needs. For example, areas in the short-exposure visible light images with brightness greater than a first brightness threshold can be designated as bright areas, and areas in the long-exposure visible light images with brightness less than a second brightness threshold can be designated as dark areas. The first and second brightness thresholds can be set based on experience or requirements.

[0043] Furthermore, in this embodiment, long-exposure near-infrared (NIR) images and short-exposure NIR images can be considered not overexposed if the following conditions are met: information in most areas of both long-exposure and short-exposure NIR images can be captured, and for the corresponding areas of overexposed areas in long-exposure visible light images in both images, the information of the near-infrared images is retained. This approach facilitates finding the spatial matching relationship between long-exposure and short-exposure frames and helps reconstruct the gradient of overexposed areas in long-exposure visible light brightness images based on the gradient of the near-infrared images. In this embodiment, bright areas in short-exposure visible light images are not overexposed to retain information about the bright areas for calculating the mapping relationship between the visible light brightness map and the near-infrared image in the bright areas of the short-exposure scene. Simultaneously, the color information of the bright areas in the short-exposure visible light images is retained for subsequent chromaticity reconstruction of the overexposed areas in the long-exposure visible light images.

[0044] In some embodiments, a long-exposure visible light image and a long-exposure near-infrared image belong to the same frame of the target scene, while a short-exposure visible light image and a short-exposure near-infrared image belong to another frame of the target scene.

[0045] In step S12, the brightness of the overexposed areas of the long-exposure visible light image is reconstructed based on the long-exposure near-infrared image, the short-exposure visible light image, and the short-exposure near-infrared image.

[0046] In step S13, chromaticity reconstruction is performed on the overexposed areas of the long-exposure visible light image based on the short-exposure visible light image.

[0047] In step S14, a high dynamic range image is generated based on the long-exposure visible light image and its brightness reconstruction results and chromaticity reconstruction results.

[0048] As described above, the method for generating a high dynamic range image according to the embodiments of this disclosure reconstructs the brightness and color of overexposed areas in a long-exposure visible light image, and generates a high dynamic range image based on the reconstruction results and the long-exposure visible light image. The resulting high dynamic range image exhibits clear imaging of both bright and dark areas without artifacts.

[0049] Figure 2 This is a flowchart illustrating brightness reconstruction of overexposed areas in a long-exposure visible light image according to an embodiment of this disclosure. For example... Figure 2 As shown, the brightness reconstruction process includes the following steps S21 to S25.

[0050] In step S21, the mapping relationship between the brightness of the short-exposure visible light image and the near-infrared (NIR) intensity of the short-exposure near-infrared image is obtained as the first mapping relationship. Specifically, there is a one-to-one correspondence between the pixels in the short-exposure visible light image and the pixels in the short-exposure near-infrared image, and each pixel has a mapping relationship to represent the relationship between the visible light brightness of the pixel and the infrared intensity of its corresponding pixel.

[0051] In step S22, the spatial matching relationship between the long-exposure near-infrared image and the short-exposure near-infrared image is obtained. Since neither the short-exposure nor the long-exposure near-infrared image is overexposed, the accurate spatial matching relationship between the long-exposure and short-exposure near-infrared images can be obtained in step S22.

[0052] In step S23, the image patch correspondence between the long-exposure near-infrared image and the short-exposure near-infrared image is obtained as a second mapping relationship based on the spatial matching relationship between the two images. Since the spatial relationships between the short-exposure near-infrared image and the short-exposure visible image are consistent, the spatial relationships between the long-exposure near-infrared image and the long-exposure visible image are also consistent. Therefore, based on the spatial matching relationships between the short-exposure near-infrared image and the short-exposure visible image, the spatial matching relationships between the long-exposure near-infrared image and the short-exposure near-infrared image, and the spatial matching relationships between the long-exposure near-infrared image and the long-exposure visible image, the image patch correspondence between the long-exposure visible image and the short-exposure visible image can be accurately obtained. For example, for a region in the long-exposure visible image... Based on the long-exposure near-infrared image and the region Image patches at the same location Obtain the closest image patch in a short-exposure near-infrared image. Then, obtain the image patch in the short-exposure visible light image. Image patches at the same location The image block That is, with the region Matched image patches.

[0053] In some embodiments, in step S23, the degree of matching of patches in the long-exposure visible light image and the short-exposure visible light image can be determined based on the L2 distance of their visible light brightness channels.

[0054] In step S24, the mapping relationship between the brightness of the long-exposure visible light image and the near-infrared intensity of the long-exposure near-infrared image is obtained based on the first and second mapping relationships and used as the third mapping relationship. For example, for an image patch... If its corresponding first mapping relationship is F, then the image patch Matching region The corresponding third mapping relationship is also F.

[0055] In step S25, the brightness of the overexposed area of ​​the long-exposure visible light image is reconstructed based on the third mapping relationship and the near-infrared intensity of the long-exposure near-infrared image.

[0056] According to one embodiment of this disclosure, reconstructing the brightness of an overexposed region in a long-exposure visible light image based on a third mapping relationship and the near-infrared intensity of the long-exposure near-infrared image includes: mapping the near-infrared intensity of the corresponding overexposed region in the long-exposure near-infrared image to brightness based on the third mapping relationship, and reconstructing the brightness of the overexposed region in the long-exposure visible light image based on the brightness mapping result. The corresponding overexposed region refers to the region in the long-exposure near-infrared image that corresponds to the overexposed region in the long-exposure visible light image.

[0057] Figure 3This diagram illustrates the brightness reconstruction of an overexposed area according to an embodiment of the present disclosure. As shown, the long-exposure near-infrared image and the long-exposure visible light image are the same size, with each pixel corresponding to the previous one. The overexposed area has the same position and size as the overexposed area. Taking pixel b1 in the long-exposure visible light image as an example, its corresponding pixel is a1. During brightness reconstruction, the brightness of b1 can be replaced with f(N_a1), where f is a mapping function corresponding to the third mapping relationship, and N_a1 represents the near-infrared intensity of pixel a1. In this way, the visible light brightness of each pixel in the overexposed area can be replaced with the mapped value of its corresponding near-infrared intensity, thereby completing the brightness reconstruction of the overexposed area.

[0058] like Figure 4 As shown, in one embodiment of this disclosure, reconstructing the brightness of the overexposed area of ​​a long-exposure visible light image based on a third mapping relationship and the near-infrared intensity of the long-exposure near-infrared image includes the following steps S41 to S43.

[0059] In step S41, the gradient of the overexposed region in the long-exposure near-infrared image is mapped to the gradient of the overexposed region in the long-exposure visible light image according to the third mapping relationship, wherein the overexposed region corresponds to the overexposed region in the long-exposure visible light image. Figure 3 For example, for pixel b1 in the overexposed region, its gradient is f(G_a1), where G_a1 represents the gradient of pixel a1 in the overexposed region. This method can be used to reconstruct the gradient of each pixel in the overexposed region.

[0060] In step S42, the gradient of the long-exposure visible light image is obtained based on the gradient of the overexposed area and the gradient of the non-overexposed area. The gradient of the non-overexposed area refers to the gradient of the visible light brightness channel in the non-exposed area of ​​the long-exposure visible light image.

[0061] In step S43, the brightness of the overexposed areas of the long-exposure visible light image is reconstructed based on the gradient of the long-exposure visible light image. Specifically, in step S43, the brightness of the overexposed areas of the visible light image can be recovered based on the gradient of the long-exposure visible light image using the Poisson editing algorithm, but this disclosure is not limited thereto.

[0062] In some embodiments, the brightness of the overexposed areas in the reconstructed long-exposure visible light image may be greater than 1. To address this issue, the method for generating a high dynamic range image in this embodiment may further include: compressing the brightness of the overexposed areas of the long-exposure visible light image to a range of 0 to 1. Specifically, the long-exposure visible light image is normalized, and then the pixel values ​​of the visible light brightness channel in the long-exposure visible light image are adjusted using the following formula (1):

[0063]

[0064] Where V i 'V' represents the adjusted visible light brightness channel pixel value of the i-th pixel in a long-exposure visible light image. i V represents the pixel value of the visible light luminance channel of the i-th pixel. max V represents the maximum pixel value of the visible light brightness channel in a long-exposure visible light image. w It can be defined by the following equation (2):

[0065]

[0066] Where k is the number of pixels in the long-exposure visible light image, and δ is a minimum value that can be set based on experience.

[0067] As can be seen from the above description, in this embodiment of the present disclosure, gradient information is filled in the gradient domain for overexposed areas in long-exposure visible light images, and brightness is reconstructed using the Poisson editing method. Furthermore, the reconstructed brightness is compressed, which helps to improve the clarity of high dynamic range images.

[0068] Figure 5A This is a flowchart illustrating the mapping relationship between the brightness of the short-exposure visible light image and the near-infrared intensity of the short-exposure near-infrared image obtained according to an embodiment of this disclosure, as a first mapping relationship. For example... Figure 5A As shown, the method for obtaining the first mapping relationship in this embodiment includes the following steps S51 and S52.

[0069] In step S51, the short-exposure scene is segmented into several regions based on the short-exposure visible light image and the short-exposure near-infrared image. In some embodiments, in step S51, the short-exposure scene can be segmented based on the grayscale values ​​of pixels in the short-exposure visible light image and the short-exposure near-infrared image. For example, Figure 5B and Figure 5C These are example images of short-exposure visible light images and short-exposure near-infrared images, respectively, from embodiments of this disclosure. Figure 5D The image shown is an example of the segmentation results for a short exposure scene. Figure 5D Each color represents a region.

[0070] In step S52, a first mapping relationship is obtained based on the brightness of the visible light image and the near-infrared intensity in each region.

[0071] In some embodiments, for any region A in a short exposure scene, the first mapping relationship corresponding to region A can be represented by the following equation (3):

[0072] V = a × N + b, Equation (3),

[0073] Where V represents the pixel value of the visible light brightness channel of the pixel in region A of the short-exposure visible light image, N represents the near-infrared intensity of the pixel in region A of the short-exposure near-infrared image, and a and b are mapping parameters that depend on the visible light brightness and near-infrared intensity of the pixel in region A and can be obtained by fitting. It should be noted that the above formula (3) applies to all pixels in region A. For example, for pixel b2 in region A of the short-exposure visible light image, its corresponding pixel in the short-exposure near-infrared image is a2, then V_b2=a×N_a2+b, where V_b2 is the pixel value of the visible light brightness channel of pixel b2 and N_a2 is the near-infrared intensity of pixel a2.

[0074] In some embodiments, the mapping parameters a and b can be obtained by the following equations (4) and (5), respectively:

[0075]

[0076]

[0077] Where M represents the number of pixels in region A, and V i N represents the pixel value of the visible light luminance channel of the i-th pixel. i This represents the near-infrared intensity of the corresponding pixel at the i-th pixel.

[0078] In some other embodiments, for any region B in a short exposure scene, the first mapping relationship corresponding to region B can be represented by the following equation (6):

[0079] V = a × N, Equation (6),

[0080] Where V represents the pixel value of the visible light brightness channel of the pixel in region B of the short-exposure visible light image, N represents the near-infrared intensity of the pixel in region B of the short-exposure near-infrared image, and a is a mapping parameter that depends on the visible light brightness and near-infrared intensity of the pixel in region B and can be obtained through fitting. In comparison, this method only requires obtaining one mapping parameter, which is convenient to calculate and requires fewer resources.

[0081] Figure 6A This is a flowchart illustrating the division of a short-exposure scene into several regions in an embodiment of this disclosure. For example... Figure 6A As shown, the process for segmenting short exposure scenes in this embodiment includes the following steps S61a to S63a.

[0082] In step S61a, each channel of the short-exposure visible light image is segmented to obtain the first segmentation result.

[0083] In step S62a, the short-exposure near-infrared image is segmented to obtain a second segmentation result.

[0084] In step S63a, the segmentation result of the short exposure scene is obtained based on the first segmentation result and the second segmentation result.

[0085] For example, if the short-exposure visible light image is a short-exposure RGB image, the method for segmenting it in this embodiment of the present disclosure is as follows: the short-exposure RGB image is converted into three channels: H, S, and V. The H channel, S channel, V channel, and short-exposure near-infrared image are segmented sequentially. That is, after the H channel is segmented, the S channel is segmented according to the segmentation result of the H channel, the V channel is segmented according to the segmentation result of the S channel, and the short-exposure near-infrared image is segmented according to the segmentation result of the V channel, thereby obtaining the final segmentation result.

[0086] Figure 6B This is a flowchart illustrating the segmentation of any channel B in a short-exposure visible light image according to an embodiment of this disclosure. For example... Figure 6B As shown, the process of segmenting channel B in this embodiment includes the following steps S61b and S62b.

[0087] In step S61b, a multi-level grayscale histogram of channel B is obtained. For example, in some embodiments, a 256-level grayscale histogram of channel B can be obtained, that is, a grayscale histogram of levels 0 to 255 is established based on the grayscale values ​​of the pixels in channel B.

[0088] In step S62b, the gray value peak of channel B is obtained from the multi-level gray value histogram of channel B, and the pixels of the channel are segmented using the valley between adjacent gray value peaks as the segmentation boundary.

[0089] Figure 7 This is a flowchart illustrating the chromaticity reconstruction of overexposed areas in a long-exposure visible light image based on a short-exposure visible light image, according to an embodiment of this disclosure. For example... Figure 7 As shown, the flowchart for chromaticity reconstruction in this embodiment includes the following steps S71 and S72.

[0090] In step S71, the corresponding patch of the overexposed area of ​​the long exposure visible light image is obtained from the short exposure visible light image.

[0091] In step S72, the chromaticity of the overexposed areas of the long-exposure visible light image is reconstructed based on the chromaticity of the corresponding patch. For example, in step S72, the chromaticity of the overexposed areas of the long-exposure visible light image can be replaced with the chromaticity of the corresponding patch, but this disclosure is not limited thereto.

[0092] In some embodiments, after chromaticity reconstruction of the overexposed areas of a long-exposure visible light image, guided filtering can be used to smooth the chromaticity channels of the long-exposure visible light image.

[0093] Figure 8A This is a flowchart illustrating the acquisition of long-exposure visible light images, long-exposure near-infrared images, short-exposure visible light images, and short-exposure near-infrared images in embodiments of this disclosure. For example... Figure 8A As shown, the method for obtaining the above-mentioned image in this embodiment includes the following steps S81 and S82.

[0094] In step S81, long-exposure raw image files and short-exposure raw image files are acquired, wherein the long-exposure raw image files and short-exposure raw image files are acquired by the image acquisition device using different exposure parameters on the target scene. In this embodiment of the present disclosure, the long-exposure raw image file can be a RAW format file acquired by the image acquisition device using a long exposure time, such as 8ms, and the short-exposure raw image file can be a RAW format file acquired by the image acquisition device using a short exposure time, such as 2ms.

[0095] In step S82, the long-exposure original image file is de-mosaiced to obtain a long-exposure visible light image and a long-exposure near-infrared image, and the short-exposure original image file is de-mosaiced to obtain a short-exposure visible light image and a short-exposure near-infrared image.

[0096] The following section will use a long-exposure raw image file containing R, G, B, and NIR channels as an example to describe the demosaic processing in detail. It should be noted that the channels of the long-exposure raw image file in this embodiment are not limited to R, G, B, and NIR.

[0097] In some embodiments, the R, G, B, and NIR channels of the long-exposure original image file can be extracted and interpolated to obtain long-exposure visible light and long-exposure near-infrared images. Specifically, the NIR channel can be extracted and synthesized into a single-channel image with a resolution of [W / 2, H / 2], which is the long-exposure near-infrared image, where W and H are the width and height of the long-exposure original image file, respectively. After averaging the G channel, the R, G, and B channels are re-merged into a standard Bayer format RAW image with a resolution of [W / 2, H / 2]. De-mosaicing of this Bayer format RAW image yields an RGB color image, i.e., the long-exposure visible light image.

[0098] In some embodiments, the long-exposure raw image file can also be processed to obtain long-exposure visible light images and long-exposure near-infrared images. Specifically, the R, G, and B channels are re-merged into a standard Bayer format RAW image, and this RAW image is de-mosaiced to obtain a down-resolution RGB color image. The NIR channel is interpolated to obtain a full-resolution NIR single-channel image, which is the long-exposure near-infrared image. The down-resolution RGB color image and the NIR single-channel image are then fused to obtain a full-resolution RGB color image, i.e., a long-exposure visible light image.

[0099] Furthermore, in this embodiment of the disclosure, a similar method described above can be used to de-mosaic the short-exposure raw image file to obtain short-exposure visible light images and short-exposure near-infrared images.

[0100] In some embodiments, the method for generating a high dynamic range image may further include: adjusting the exposure parameters of the image acquisition device to ensure that neither the short-exposure visible light image nor the long-exposure near-infrared image is overexposed, and that the bright areas of the short-exposure visible light image are not overexposed. Specifically, after acquiring the long-exposure visible light image, the long-exposure near-infrared image, the short-exposure visible light image, and the short-exposure near-infrared image, it is determined whether the four images meet preset conditions. If not, the exposure parameters of the image acquisition device are adjusted, and the original long-exposure image file and the original short-exposure image file are re-acquired until the acquired long-exposure visible light image, the long-exposure near-infrared image, the short-exposure visible light image, and the short-exposure near-infrared image meet the preset conditions. The preset conditions may, for example, be: neither the short-exposure visible light image nor the long-exposure near-infrared image is overexposed, the bright areas of the short-exposure visible light image are not overexposed, and the dark areas of the long-exposure visible light image are clearly imaged.

[0101] In some embodiments, the array of the image acquisition device includes four channels: R, G, B, and NIR, and the spatial relationship between the RGB channels and the NIR channels in the same frame image acquired by the image acquisition device is basically consistent. Figure 8B and Figure 8C Two array arrangements of image acquisition devices are shown, but this disclosure is not limited thereto.

[0102] It should be noted that this disclosure does not limit the acquisition methods for long-exposure visible light images, long-exposure near-infrared images, short-exposure visible light images, and short-exposure near-infrared images. For example, in some embodiments, these images can be acquired in real time using the aforementioned image acquisition device or a similar device. In other embodiments, they can be visible light images and near-infrared images that meet preset conditions and have consistent spatial relationships under long and short exposures, acquired from other devices. Furthermore, the image acquisition device can be an independent acquisition device or part of the execution body of the method for generating high dynamic range images provided in the embodiments of this disclosure. That is, the image acquisition device and the electronic device executing the method provided in the embodiments of this disclosure can be the same device or different devices. When the visible light images and near-infrared images are images that meet the requirements and are acquired from other devices, de-mosaicing can be performed according to the specific array arrangement of the acquired original files.

[0103] This disclosure also provides an apparatus for generating high dynamic range images. Figure 9 This is a schematic diagram illustrating the structure of an apparatus 100 for generating high dynamic range images according to an embodiment of the present disclosure. Figure 9 As shown, the apparatus 100 provided in this embodiment includes an image acquisition module 101, a brightness reconstruction module 102, a chromaticity reconstruction module 103, and an image reconstruction module 104. The image acquisition module 101 is configured to acquire long-exposure visible light images, long-exposure near-infrared images, short-exposure visible light images, and short-exposure near-infrared images of a target scene. The short-exposure near-infrared images and long-exposure near-infrared images are not overexposed, the bright areas of the short-exposure visible light images are not overexposed, and the spatial relationships of the long-exposure visible light images and long-exposure near-infrared images are consistent. The brightness reconstruction module 102 is configured to reconstruct the brightness of overexposed areas in the long-exposure visible light images based on the long-exposure near-infrared images, short-exposure visible light images, and short-exposure near-infrared images. The chromaticity reconstruction module 103 is configured to reconstruct the chromaticity of overexposed areas in the long-exposure visible light images based on the short-exposure visible light images. The image reconstruction module 104 is configured to generate a high dynamic range image based on the long-exposure visible light images and their brightness and chromaticity reconstruction results.

[0104] It should be noted that the modules in the apparatus 100 for generating high dynamic range images in this embodiment of the present disclosure are related to... Figure 1 The steps S11 to S14 in the method for generating high dynamic range images shown correspond one-to-one, and will not be elaborated further here.

[0105] This disclosure also provides a computer-readable storage medium having a computer program stored thereon. The computer program is executed to implement the method for generating high dynamic range images according to any embodiment of this disclosure.

[0106] In this disclosure, any combination of one or more storage media may be used. The storage medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0107] This disclosure also provides an electronic device. Figure 10 The diagram shown is a structural schematic of an electronic device 200 according to an embodiment of this disclosure. Figure 10 As shown, in this embodiment, the electronic device 200 includes a memory 201 and a processor 202.

[0108] The memory 201 is configured to store computer programs. In some embodiments, the memory 201 includes various media capable of storing program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card, or optical disk.

[0109] Specifically, memory 201 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Electronic device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 201 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0110] The processor 202 is connected to the memory 201 and is configured to execute a computer program stored in the memory 201 to cause the electronic device 200 to perform the method for generating high dynamic range images provided in the embodiments of this disclosure.

[0111] In some embodiments, processor 202 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. Processor 202 may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0112] In some embodiments, the electronic device 200 may further include a display 203. The display 203 is communicatively connected to the memory 201 and the processor 202, and is used to display a GUI interface related to the method of generating high dynamic range images.

[0113] The scope of protection of the method for generating high dynamic range images described in this disclosure is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this disclosure is included within the scope of protection of this disclosure.

[0114] This disclosure also provides an apparatus for generating high dynamic range (HDR) images. The apparatus for generating HDR images can implement the method for generating HDR images described in this disclosure. However, the apparatus for implementing the method for generating HDR images described in this disclosure includes, but is not limited to, the structure of the apparatus for generating HDR images listed in this embodiment. Any structural modifications and substitutions of the prior art made based on the principles of this disclosure are included within the protection scope of this disclosure.

[0115] In summary, the method for generating high dynamic range (HDR) images provided in this disclosure reconstructs the brightness and color of overexposed areas in a long-exposure visible light image, and generates an HDR image based on the reconstruction results and the long-exposure visible light image. The resulting HDR image exhibits clear imaging of both bright and dark areas without artifacts. Therefore, this disclosure effectively overcomes the various shortcomings of the prior art and possesses high industrial applicability.

[0116] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the claims of this disclosure.

Claims

1. A method of generating a high dynamic range image, characterized by, The method comprises: obtaining a long-exposure visible light image, a long-exposure near-infrared image, a short-exposure visible light image and a short-exposure near-infrared image of a target scene, wherein the short-exposure near-infrared image and the long-exposure near-infrared image are both underexposed, a bright area in the short-exposure visible light image is underexposed, the long-exposure visible light image and the long-exposure near-infrared image have a consistent spatial relationship, and the short-exposure visible light image and the short-exposure near-infrared image have a consistent spatial relationship; reconstructing luminance of an overexposed area of the long-exposure visible light image according to the long-exposure near-infrared image, the short-exposure visible light image and the short-exposure near-infrared image; reconstructing chrominance of the overexposed area of the long-exposure visible light image according to the short-exposure visible light image; and generating a high dynamic range image according to the long-exposure visible light image and the reconstructed luminance and chrominance thereof. The method of reconstructing luminance of the overexposed area of the long-exposure visible light image according to the long-exposure near-infrared image, the short-exposure visible light image and the short-exposure near-infrared image comprises:

2. The method of claim 1, wherein, obtaining a mapping relationship between luminance of the short-exposure visible light image and near-infrared intensity of the short-exposure near-infrared image as a first mapping relationship; obtaining a spatial matching relationship between the long-exposure near-infrared image and the short-exposure near-infrared image; obtaining a corresponding relationship of image blocks of the long-exposure visible light image and the short-exposure visible light image as a second mapping relationship according to the spatial matching relationship between the long-exposure near-infrared image and the short-exposure near-infrared image; obtaining a mapping relationship between luminance of the long-exposure visible light image and near-infrared intensity of the long-exposure near-infrared image as a third mapping relationship according to the first mapping relationship and the second mapping relationship; and reconstructing luminance of the overexposed area of the long-exposure visible light image according to the third mapping relationship and the near-infrared intensity of the long-exposure near-infrared image. The method of reconstructing luminance of the overexposed area of the long-exposure visible light image comprises:

3. The method of claim 2, wherein, mapping a gradient of an overexposed corresponding area in the long-exposure near-infrared image to a gradient of the overexposed area of the long-exposure visible light image according to the third mapping relationship, wherein the overexposed corresponding area corresponds to the overexposed area of the long-exposure visible light image; obtaining a gradient of the long-exposure visible light image according to the gradient of the overexposed area and the gradient of a non-overexposed area of the long-exposure visible light image; and reconstructing luminance of the overexposed area of the long-exposure visible light image according to the gradient of the long-exposure visible light image. The method of obtaining a mapping relationship between luminance of the short-exposure visible light image and near-infrared intensity of the short-exposure near-infrared image as a first mapping relationship comprises:

4. The method of claim 2, wherein, segmenting a short-exposure scene into a plurality of areas according to the short-exposure visible light image and the short-exposure near-infrared image; and obtaining the first mapping relationship according to luminance of a visible light image and near-infrared intensity in each area. For an area, the first mapping relationship corresponding to the area is as follows: V = a × N + b, wherein V represents visible light luminance of a pixel point in the area, N represents near-infrared intensity of the pixel point in the area, a and b are mapping parameters, and the mapping parameters depend on the visible light luminance and the near-infrared intensity of the pixel point in the area.

5. The method of claim 4, wherein, ​ 6. The method of claim 4, wherein, For a region, a first mapping relationship corresponding to the region is as follows: V=a*N, where V represents the visible light brightness of a pixel point in the region, N represents the near-infrared intensity of the pixel point in the region, and a is a mapping parameter, which depends on the visible light brightness and the near-infrared intensity of the pixel point in the region.

7. The method of claim 4, wherein, The short-exposure scene is segmented into a plurality of regions, including: segmenting each channel of the short-exposure visible light image to obtain a first segmentation result; segmenting the short-exposure near-infrared image to obtain a second segmentation result; and obtaining a segmentation result of the short-exposure scene according to the first segmentation result and the second segmentation result.

8. The method of claim 7, wherein, Segmenting each channel of the short-exposure visible light image includes: obtaining a multi-level gray histogram of a channel; and obtaining a gray peak value of the channel in the multi-level gray histogram, and segmenting the pixel points of the channel by taking a valley between adjacent gray peak values as a segmentation boundary.

9. The method of claim 1, wherein, The chroma reconstruction of the overexposed region of the long-exposure visible light image according to the short-exposure visible light image includes: in the short-exposure visible light image, obtaining a corresponding tile of the overexposed region of the long-exposure visible light image; and reconstructing the chroma of the overexposed region of the long-exposure visible light image according to the chroma of the corresponding tile.

10. The method of claim 1, wherein, The long-exposure visible light image, the long-exposure near-infrared image, the short-exposure visible light image, and the short-exposure near-infrared image are obtained, including: obtaining a long-exposure original image file and a short-exposure original image file, which are obtained by an image acquisition device from a target scene by using different exposure parameters; and performing demosaicing on the long-exposure original image file to obtain the long-exposure visible light image and the long-exposure near-infrared image, and performing demosaicing on the short-exposure original image file to obtain the short-exposure visible light image and the short-exposure near-infrared image.

11. The method of claim 10, wherein, Further comprising: adjusting the exposure parameters of the image acquisition device, so that the short-exposure visible light image and the long-exposure near-infrared image are not overexposed, and the bright region of the short-exposure visible light image is not overexposed.

12. The method of claim 1, wherein, The short-exposure near-infrared image and the long-exposure near-infrared image are not overexposed, so that the information of most regions in the short-exposure near-infrared image and the long-exposure near-infrared image can be captured, and for the overexposed region in the long-exposure visible light image, near-infrared image information is retained in the corresponding region in the short-exposure near-infrared image and the long-exposure near-infrared image; the bright region of the short-exposure visible light image is not overexposed, so that the brightness information and the chroma information of the bright region are retained.

13. An apparatus for generating a high dynamic range image, characterized by including: an image acquisition module configured to obtain a long-exposure visible light image, a long-exposure near-infrared image, a short-exposure visible light image, and a short-exposure near-infrared image of a target scene, wherein the short-exposure near-infrared image and the long-exposure near-infrared image are not overexposed, the bright region of the short-exposure visible light image is not overexposed, the spatial relationship of the long-exposure visible light image and the long-exposure near-infrared image is consistent, and the spatial relationship of the short-exposure visible light image and the short-exposure near-infrared image is consistent; a luminance reconstruction module configured to perform luminance reconstruction on the overexposed region of the long-exposure visible light image according to the long-exposure near-infrared image, the short-exposure visible light image and the short-exposure near-infrared image; a chrominance reconstruction module configured to perform chrominance reconstruction on the overexposed region of the long-exposure visible light image according to the short-exposure visible light image; and an image reconstruction module configured to generate a high dynamic range image according to the long-exposure visible light image and the luminance reconstruction result and the chrominance reconstruction result thereof.

14. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed to implement the method according to any one of claims 1 to 12.

15. An electronic device, comprising: comprising: a memory configured to store a computer program; and a processor configured to invoke the computer program to execute the method according to any one of claims 1 to 12.

Images