Depth high dynamic range imaging based on wavelet transform

Abstract

An image processing apparatus (700) comprising one or more processors (704) is described herein for: receiving (601) a plurality of input images (301, 302, 303); for each input image, forming (602) a decomposition dataset by decomposing the input image (301, 302, 303) or a filtered version thereof (307, 308, 309) into a plurality of frequency-specific components (313), each frequency-specific component representing occurrences of features of a respective frequency interval in the input image or filtered version thereof; processing (603) each decomposition dataset using one or more convolutional neural networks to form a combined image dataset (327); performing (604) a build operation on the combined image dataset (327), the build operation being adapted for image building from a plurality of frequency-specific components, thereby forming an output image (333) representing a combination of the input images. The generated HDR output image can have fewer artifacts and provide better quality results. Furthermore, the apparatus is computationally efficient, maintaining a good balance between accuracy and efficiency.

Description

TECHNICAL FIELD

[0001] The present invention relates to the generation of High Dynamic Range (HDR) images. BACKGROUND

[0002] HDR imaging using multiple Low Dynamic Range (LDR) images is a technique in computational photography for generating high quality HDR images with a large range of luminance by exploiting information from multiple LDR images.

[0003] Digital cameras can typically capture LDR images with a limited range of luminance at once, where some overexposed and / or underexposed regions exist, thus reducing the image quality. For cameras used in wearable devices, there are limitations on the sensor and aperture, so the number of electrons reaching each pixel is limited, making it difficult to capture an HDR image at once. Existing high-end digital devices use large sensors and large apertures to capture HDR images, which makes them difficult to integrate into wearable devices such as smartphones. These large devices are also typically very expensive.

[0004] A practical solution is to capture several LDR images with different exposure times and merge them into a single HDR image. To generate an HDR image, an HDR imaging method should be able to recover the missing information (overexposed and underexposed regions) from multiple LDR images and preserve the useful information.

[0005] Figure 1 A typical HDR imaging framework is shown, where the input is three LDR images 101-103 (101 is short exposure, 102 is medium exposure, and 103 is long exposure), and the output is a high quality HDR image 104. The medium exposure image 102 (input 2) is considered as the reference image, while the short exposure image 101 and the long exposure image 103 (inputs 1 and 3, respectively) are the supporting images. It is important to note that there is camera motion and object motion between the reference and supporting images.

[0006] Existing methods in this field suffer from different types of artifacts, including ghosting, loss of details, color washout, and noise.

[0007] Preferably, a good HDR imaging method should meet the following requirements: The method should generate high-quality HDR images. Ideally, the generated HDR images should not have overexposed or underexposed areas, fully utilizing supplementary information from different LDR images with varying exposures. The HDR image should also have good detail, accurate colors, and be noise-free throughout. The generated image should be free of ghosting. Camera and object motion are often present between the captured LDR images. If these LDR images are not well aligned, ghosting artifacts may appear in the final merged HDR image. Preferably, the method should also be computationally efficient, allowing for effective application in wearable devices.

[0008] Due to the strong representational capabilities of deep convolutional neural networks (CNNs), an increasing number of HDR imaging algorithms have been proposed that combine them with deep neural networks. In the method described by Nima KhademiKalantari and Ravi Ramamoorthi in their ACM Transactions on Graphics (2017), "Deep high dynamic range imaging of dynamic scenes," a support image is aligned with a reference image using optical flow. Then, a CNN network is used to merge the aligned LDR images to generate an HDR image. The method described in the paper "Towards practical and efficient high-resolution HDR deghosting with CNN" (2020) published in the proceedings of the European Conference on Computer Vision (ECCV) by K. Ram Prabhakark, Susmit Agrawal, Durgesh Kumar Singh, Balraj Ashwath, and R. Venkatesh Babu merges low-resolution input LDR images to save computational costs and uses a bilateral guided upsampler to restore the images to their original scale. These methods then require optical flow to align frames, which incurs additional computational costs, and some hardware does not support the warping operation, meaning that the method cannot be applied to such hardware.

[0009] To avoid using optical flow, the method described in the paper "Deep high dynamic range imaging with large foreground motions" (2018) published in the European Conference on Computer Vision (ECCV) proceedings by Shangzhe Wu, Jiarui Xu, Yu-Wing Tai, and Chi-Keung Tang formulates the HDR imaging problem as an image-to-image transformation problem. This method uses a UNet-structured network to directly map the input LDR image to the HDR image. However, it exhibits worse performance than algorithms using optical flow. The method described in the paper "Multi-scale dense networks for deep high dynamic range imaging" published at the 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) by Qingsen Yan, Dong Gong, Pingping Zhang, Qingfeng Shi, Jinqiu Sun, Ian Reid, and Yanning Zhang employs three sub-networks of different scales to progressively reconstruct the HDR image. The paper "Deep HDR imaging via a non-local network" (2020) published in *IEEE Transactions on Image Processing* by Qingsen Yan, Lei Zhang, Yu Liu, Yu Zhu, Jinqiu Sun, Qinfeng Shi, and Yanning Zhang describes a method called NHDRRnet. This method uses a UNet to extract low-dimensional features, which are then fed into a global non-local network that fuses features from the input based on their correspondences. This method effectively removes ghosting artifacts from the final output. However, its performance is still limited by the traditional UNet architecture and is worse than previous methods using optical flow.

[0010] The paper "Attention-guided network for ghost-free high dynamic range imaging" (2019) published in the proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) by Qingsen Yan, Dong Gong, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, Ian Reid, and Yanning Zhang describes a method that proposes a fixed-scale network with an attention module to handle misaligned regions in support frames and uses expanded residual dense blocks to merge features into a single HDR image. While this method achieves state-of-the-art results, its computational efficiency is low, making it difficult to apply to wearable devices.

[0011] A method needs to be developed to overcome these problems. Summary of the Invention

[0012] According to one aspect, the present invention provides an image processing apparatus comprising one or more processors, for: receiving a plurality of input images; for each input image, forming a decomposed dataset by decomposing the input image or a filtered version thereof into a plurality of frequency-specific components, each frequency-specific component representing the occurrence of a feature at a corresponding frequency interval in the input image or a filtered version thereof; processing each decomposed dataset using one or more convolutional neural networks to form a combined image dataset; performing a construction operation on the combined image dataset, the construction operation being adapted to construct images from the plurality of frequency-specific components, thereby forming an output image representing the combination of the input images.

[0013] Compared to previous methods, the generated HDR output images may have fewer artifacts and provide better quality results. Furthermore, the device is computationally efficient, maintaining a good balance between accuracy and efficiency.

[0014] The step of decomposing the input image may include performing a discrete wavelet transform (DWT) operation on the input image. This may help reduce information loss during downsampling.

[0015] The construction operation can be an inverse discrete wavelet transform (IDWT) operation. The signal can then be reconstructed using the output of the DWT.

[0016] The device may include a camera or other imaging device. The device may be configured to respond to input from a user of the device, causing the camera or imaging device to capture a plurality of input images, each of which is captured with a different exposure than the other input images. This allows the generation of a high-quality HDR image with a wide range of brightness by utilizing information from multiple LDR images.

[0017] The decomposed data can be formed by decomposing the corresponding input image versions filtered by convolutional filters. This allows adjustment of the number of channels in the input, thereby enabling the formation of feature maps for each input image.

[0018] The apparatus can be used to: mask and weight at least some regions of the decomposed datasets to form attention-filtered decomposed data; select a subset of components of the attention-filtered decomposed data, wherein the components correspond to frequencies lower than other components of the attention-filtered decomposed data; and merge at least some components in the subset to form merged data, wherein the merged data constitutes the input to the construction operation. This allows the merging of the low-frequency components.

[0019] The apparatus can be used to: decompose the attention-filtered data; merge the relatively low-frequency components of the attention-filtered data through multiple residual operations to form convolutional low-frequency data; and perform a reconstruction operation based on the relatively high-frequency components of the attention-filtered data and the convolutional low-frequency data. This allows the low-frequency components to be merged and the high-frequency components to be used for upsampling.

[0020] The apparatus can be used to: for each input image, form a corresponding decomposed dataset by decomposing the input image or a filtered version thereof into a first plurality of frequency-specific component sets, each frequency-specific component set representing the occurrence of features at a corresponding frequency interval in the input image or a filtered version thereof; perform a convolution operation on each of the frequency-specific component sets to form convolutional data; and decompose the convolutional data into a second plurality of frequency-specific component sets, each frequency-specific component set representing the occurrence of features at a corresponding frequency interval in the convolutional data. This allows downsampling of each input image before merging.

[0021] The apparatus can be used to: merge a first subset of the second plurality of frequency-specific component sets to form first merged data; perform masking and weighted combination on the first subset of the second plurality of frequency-specific component sets and the first merged data to form first combined data; perform a first convolutional combination on a second subset of the second plurality of frequency-specific component sets to form second combined data; upsample the first combined data and the second combined data to form first upsampled data; perform masking and weighted combination on the first subset of the first plurality of frequency-specific component sets and the first upsampled data to form third combined data; perform a second convolutional combination on the second subset of the first plurality of frequency-specific component sets to form fourth combined data; and upsample the third combined data and the fourth combined data to form second upsampled data; wherein the output image is formed based on the second upsampled data. This allows for the separate fusion of several components (e.g., low-frequency components and high-frequency components) from the different input images with different frequency intervals.

[0022] The first subset can be a subset of relatively low-frequency components. The second subset can be a subset of relatively high-frequency components. These components can be generated by using a discrete wavelet transform operation. Low-frequency components contain more structural information. Therefore, using low-frequency components in the merging stage can help reduce ghosting artifacts and recover underexposed and overexposed areas of the input. High-frequency components can preserve detail information, which helps to reconstruct details during the upsampling process.

[0023] The output image can be formed based on a combination of the second upsampled data and a convolutional version of the input image. This can be achieved using global residual connections. This can enhance the representational power of the network.

[0024] According to a second aspect, the present invention provides a computer-implemented image processing method, comprising: receiving a plurality of input images; for each input image, forming a decomposed dataset by decomposing the input image or a filtered version thereof into a plurality of frequency-specific components, each frequency-specific component representing the occurrence of features at a corresponding frequency interval in the input image or a filtered version thereof; processing each decomposed dataset using one or more convolutional neural networks to form a combined image dataset; performing a construction operation on the combined image dataset, the construction operation being adapted to construct images from the plurality of frequency-specific components, thereby forming an output image representing the combination of the input images.

[0025] Compared to previous methods, HDR images generated using this method may have fewer artifacts and provide better quality results. Furthermore, the method is computationally efficient, maintaining a good balance between accuracy and efficiency.

[0026] The multiple input images may include a reference image and multiple support images, each support image having a longer or shorter exposure than the reference image. This can produce high-quality HDR images. Camera motion and object motion may exist between the reference image and the support images. Attached Figure Description

[0027] The invention will now be described by way of example with reference to the accompanying drawings.

[0028] In the attached diagram:

[0029] Figure 1 A schematic diagram of an HDR imaging task that receives input from multiple frames is shown.

[0030] Figure 2 The discrete wavelet transform (DWT) is illustrated schematically.

[0031] Figure 3 A schematic diagram of the overall network structure is shown;

[0032] Figure 4 A schematic diagram of the feature merging module is shown;

[0033] Figure 5 A schematic diagram of the upsampling module is shown;

[0034] Figure 6 An example of a computer-implemented image processing method is shown;

[0035] Figure 7 An example of a device for implementing the methods described herein is shown;

[0036] Figure 8 A qualitative comparison is shown between results generated using the methods described herein and baseline methods. Detailed Implementation

[0037] The network implemented in the apparatus and method described in this paper is UNet (as described in the paper "U-net: Convolutional networks for biomedical image segmentation" by Olaf Ronneberger, Philipp Fischer, and Thomas Brox at the International Conference on Computational Medical Imaging and Computer-Aided Interventions (Springer, Cham, 2015, pp. 234-241)), a deep neural network architecture commonly used for image processing operations such as image denoising, super-resolution, joint denoising, and demosaicing.

[0038] Wavelet transform is a useful tool for transforming images into groups of different frequency components. For example... Figure 2 As shown, wavelet transform can decompose the signal corresponding to the image into components with different frequency intervals, called low-low (LL) (shown at 201), high-low (HL) (shown at 202), low-high (LH) (shown at 203), and high-high (HH) (shown at 204).

[0039] In a preferred embodiment, the apparatus and method described herein combine wavelet transform with a UNet for HDR imaging using multiple LDR images as input. Discrete wavelet transform (DWT) and inverse discrete wavelet transform (IDWT) are used instead of max pooling / stride convolution operations in the UNet network. Using DWT reduces information loss during downsampling. IDWT can reconstruct the signal using the output of DWT to restore the feature map to the original scale.

[0040] Wavelet transform is also used in the feature merging module to merge the low-frequency components of the input. In the feature merging module, which takes the low-frequency components as input, a spatial attention module is used to process misaligned regions between the reference image and the support image.

[0041] Figure 3 The overall network architecture of one embodiment of the present invention is shown. In addition to the basic UNet structure, the network also includes a feature merging module and two upsampling modules.

[0042] Each LDR input is sent to a single encoder with the same architecture. In this example, three LDR inputs 301, 302, and 303 are used. For each input, convolutional layers 304, 305, and 306 are used to adjust the number of channels of the input and form feature maps 307, 308, and 309.

[0043] Then, as shown at 310, 311, and 312, the feature map of each input is decomposed into one low-frequency component and three high-frequency components using DWT, which are shown in the dashed box at 313. Therefore, for each input image, the decomposed dataset is formed by decomposing the feature map into multiple specific frequency components, each frequency-specific component representing the presence of features at a corresponding frequency interval in the feature map.

[0044] Then, as described in detail below, each decomposed dataset is processed using one or more convolutional neural networks to form a combined image dataset. A construction operation is performed on the combined image dataset, which is adapted to construct images from multiple frequency-specific components, thereby forming an output image representing the combination of the input images.

[0045] exist Figure 3 In the example shown, the decomposed data is formed by decomposing versions 307, 308, and 309 of the corresponding input images 301, 302, and 303 filtered by convolutional filters 304, 305, and 306. Each corresponding decomposed dataset is formed by decomposing the filtered image into a first plurality of frequency-specific component sets, each frequency-specific component set representing the occurrence of features at a corresponding frequency interval in the filtered image (feature map).

[0046] like Figure 3 As shown, in this example, while the high-frequency components are retained for later stages (as shown by dashed line 314), the low-frequency components of each set of decomposed data are used for feature extraction through another convolutional layer 315, 316, 317.

[0047] Convolutional data 318, 319, and 320 are generated respectively. Subsequently, the convolutional data is decomposed into a second set of frequency-specific components, each representing the occurrence of features at a corresponding frequency interval in the convolutional data, as shown in dashed box 324. In this example, DWT is used again, as shown in 321, 322, and 323, to decompose each input feature map 318, 319, and 320 into different frequency components (three high-frequency and one low-frequency) shown in dashed box 324. The low-frequency component is then sent from this dashed box to the feature merging module 325 (see later). Figure 4 (Description), In this feature merging module, feature fusion is performed and the high-frequency components are stored for use in later stages (as shown by dashed line 326).

[0048] The fused feature map 327, along with pre-stored low-frequency and high-frequency components, is sequentially sent to upsampling modules 328 and 329 to reconstruct the feature map at the original scale, as shown in 330. In this example, global residual connections are added (as shown in 331) to enhance the network's representational capabilities. The final feature map is shown in 332, and the tone-mapped HDR image is shown in 333.

[0049] Figure 4 The feature merging module is shown. Figure 3An exemplary illustration of (325) is shown. The input to the merging module is three low-frequency components from different LDR images, as shown at 401, 402, and 403. Here, L2 is the low-frequency component of the reference frame. L1 and L3 are the low-frequency components of the support frame.

[0050] The apparatus is used to mask and weight at least some regions of the decomposed datasets to form attention-filtered decomposed data. A subset of components of the attention-filtered decomposed data is selected, wherein the components correspond to frequencies lower than other components of the attention-filtered decomposed data. This subset is merged to form merged data, which constitutes the input to the construction operation.

[0051] In the initial stage, the inputs L1 and L3 are sent together with the reference low-frequency component L2 to the attention modules 404 and 405 to generate corresponding attention masks. and Then, the mask is applied to the corresponding inputs using element-wise multiplication to obtain L′1 and L′3:

[0052]

[0053]

[0054] As shown at 406, L′1, L2, and L′3 are concatenated and passed through a convolutional layer to compress the number of channels in feature map 407. Then, DWT is used to decompose feature map 407 into different frequency components. The high-frequency components are shown in dashed box 408. The low-frequency components 409 are passed through several residual blocks (as shown at 410) to merge the features into feature map 411. Finally, an IDWT layer 412 is used to restore the feature map to its original scale. The resulting feature map is shown at 413.

[0055] Therefore, in this example, the apparatus is also used to: decompose the attention-filtered data; merge the relative low-frequency components of the attention-filtered data through multiple residual operations (the residual block shown at 410) to form convolved low-frequency data; and perform a reconstruction operation based on the relative high-frequency components of the attention-filtered data and the convolved low-frequency data.

[0056] In this example, for each input image, the apparatus is configured to: form a corresponding decomposed dataset by decomposing the filtered feature map into a first plurality of frequency-specific component sets, each frequency-specific component set representing the occurrence of a feature at a corresponding frequency interval in the feature map; perform a convolution operation on each of the frequency-specific component sets to form convolutional data; and decompose the convolutional data into a second plurality of frequency-specific component sets, each frequency-specific component set representing the occurrence of a feature at a corresponding frequency interval in the convolutional data.

[0057] A first subset of the second plurality of frequency-specific component sets is merged to form first merged data. A masking and weighted combination is performed on the first subset of the second plurality of frequency-specific component sets and the first merged data to form first combined data. A first convolutional combination is performed on a second subset of the second plurality of frequency-specific component sets to form second combined data. The first combined data and the second combined data are upsampled to form first upsampled data. A masking and weighted combination is performed on the first subset of the first plurality of frequency-specific component sets (corresponding to the relatively low-frequency components) and the first upsampled data to form third combined data.

[0058] A second convolutional combination is performed on a second subset of the first plurality of frequency-specific component sets (corresponding to the relatively high-frequency components) to form a fourth combined data.

[0059] The third and fourth combined data are upsampled to form second upsampled data, and this data is used as... Figure 3 The result of the global residual connection 331 shown is that the output image is formed based on a combination of the second upsampled data and the convolutional version of the input image.

[0060] Figure 5 An exemplary illustration of the upsampling module is shown. Unlike previous computer vision tasks that used wavelet transforms and had only a single input (e.g., denoising, classification), HDR imaging tasks typically have multiple input LDR images. Therefore, after each DWT layer, multiple sets of low-component and high-component images can be implemented, for example:

[0061]

[0062]

[0063]

[0064] Here, L n For the nth th One input, [LL] n ,LH n HL n ,HHn [For the nth] th The input consists of components with different frequency intervals. Then, when upsampling the feature map using IDWT, only one set of these components is used. In the method described herein, a learnable merging module merges these low-frequency and high-frequency components into a set that can be used during upsampling. First, for the high-frequency components, the components with the same frequency interval are grouped together:

[0065] LHs = Concat(LH1, LH2, LH3) (6)

[0066] HLs = Concat(HL1, HL2, HL3) (7)

[0067] HHs = Concat(HH1, HH2, HH3) (8)

[0068] The components of these groups are respectively in Figure 5 The components are shown at positions 501, 502, and 503. Then, the components of these groups are passed through several convolutional layers to generate a set of high-frequency components, shown at positions 504, 505, and 506, and concentrated at position 507.

[0069] Following the steps in the feature merging module, LL1 and LL3 are sent together with the reference low-frequency component LL2 to the attention modules 511 and 512 to generate corresponding attention masks. and Furthermore, the mask is applied to the corresponding input using element-wise multiplication to obtain L′1 and L′3), and the low-frequency component LL shown at positions 508, 509, and 510 is fused. n Then, it is concatenated with the feature map F′513 from the previous layer to generate a single low-frequency component as shown at 514. Then, as shown at 515, IDWT is used to restore the feature map to the original scale using the generated low-frequency component 514 and high-frequency component 507.

[0070] As described above, for the input of the network, a set of LDR images {L1, L2, L3+} is used. Based on gamma correction, this set of LDR images is mapped to the HDR domain {H1, H2, H3+}.

[0071]

[0072] Where γ is the gamma correction parameter, t i For LDR image L i The exposure time. These two sets of images are concatenated to generate the final input set {X1,X2,X3+:}

[0073] X i=Concat(L i H i (10)

[0074] The method described in this paper formulates the HDR imaging problem as an image-to-image transformation problem. Therefore, by minimizing the predicted tone mapping in the output... The network is trained using the L1 loss between the ground truth T(H) after tone mapping and the ground truth T(H):

[0075]

[0076] Where Τ(·) is the μ-law function, which can be expressed as:

[0077]

[0078] Where μ is a coefficient that controls compression.

[0079] Therefore, the method described in this paper constitutes a learning-based approach that combines wavelet transform for HDR image fusion from multi-frame inputs.

[0080] As described above, the apparatus includes: a wavelet transform module for decomposing the input into multiple components with different frequency intervals in the feature space; a parameterized learning module for processing feature fusion of the low-frequency components and the high-frequency components in the network; and an attention module for processing unaligned regions in the support frame.

[0081] The method described in this article does not require optical flow.

[0082] The learnable module is used to individually fuse several components (low-frequency and high-frequency components) from the different input images with different frequency intervals. For the feature merging module, only the low-frequency components are used. Low-frequency components contain more structural information. Therefore, using low-frequency components in the merging stage can help reduce ghosting artifacts and recover underexposed and overexposed areas of the input. High-frequency components can preserve detail information, which helps reconstruct details during upsampling.

[0083] Figure 6An example of a computer-implemented image processing method is shown; in step 601, the method includes receiving a plurality of input images. In step 602, the method includes, for each input image, forming a decomposed dataset by decomposing the input image or a filtered version thereof into a plurality of frequency-specific components, each frequency-specific component representing the occurrence of a feature at a corresponding frequency interval in the input image or a filtered version thereof. In step 603, the method includes: processing each decomposed dataset using one or more convolutional neural networks to form a combined image dataset. In step 604, the method includes: performing a construction operation on the combined image dataset, the construction operation being adapted to construct images from the plurality of frequency-specific components, thereby forming an output image representing the combination of the input images.

[0084] The device may include an imaging device, such as a camera. The apparatus may be used to respond to input from a user of the apparatus, causing the camera or other imaging device to capture each input image at an exposure different from other input images.

[0085] Figure 7 An example of an apparatus 700 including an imaging device 701 is shown, the apparatus being used to process image data captured by at least one image sensor in the apparatus using the methods described herein. The apparatus 701 includes image sensors 702, 703. Such an apparatus 701 includes some onboard processing capabilities. This may be provided by the processor 704. The processor 704 may also be used for the basic functions of the apparatus.

[0086] Transceiver 705 is capable of communicating with other entities 710, 711 via a network. These entities may be physically located away from device 701. The network may be a publicly accessible network, such as the Internet. Entities 710, 711 may be cloud-based. Entity 710 is a computing entity. Entity 711 is a command and control entity. These entities are logical entities. In practice, each of them may be provided by one or more physical devices (e.g., servers and data storage areas), and the functionality of two or more of the entities may be provided by a single physical device. Each physical device implementing an entity includes a processor and memory. The device also includes a transceiver for sending and receiving data to and from transceiver 705 of device 701. The memory stores code in a non-transitory manner, which can be executed by the processor to implement the corresponding entity in the manner described herein.

[0087] The command and control entity 711 can train the model used in the device. This is typically a computationally intensive task, even though the obtained model can be efficiently described. Therefore, the development of the algorithm can be efficiently performed in the cloud, where abundant energy and computing resources are readily available. This is predictably more efficient than forming such a model on a typical imaging device.

[0088] In one embodiment, after the algorithm is developed in the cloud, the command and control entity can automatically generate a corresponding model and transmit it to the relevant imaging device. In this example, the model is implemented at the device 701 by the processor 704.

[0089] In another possible implementation, images can be captured by one or both of the sensors 702 and 703, and the transceiver 705 can send the image data to the cloud for processing. The generated image can then be sent back to the device 701, such as... Figure 7 As shown in 712.

[0090] Therefore, the method can be deployed in various ways, such as in the cloud, on the device, or in dedicated hardware. As mentioned above, cloud facilities can perform training to develop new algorithms or improve existing ones. Depending on the computing power near the data corpus, the training can be performed close to the source data or in the cloud, for example, using an inference engine.

[0091] Unlike existing methods, the method described in this paper does not require optical flow to align the input frames. DWT is advantageously used to reduce information loss due to max pooling and strided convolution operations. DWT is advantageously used to perform downsampling, while IDWT is used for upsampling.

[0092] Eliminating optical flow can save computational costs. Furthermore, some hardware does not support optical flow. Therefore, the method described in this paper can be used on a wider range of hardware and can be deployed in mobile devices.

[0093] By employing wavelet transforms, information loss during downsampling is reduced. Therefore, higher-quality HDR images can be generated. Compared to previous methods, this invention achieves a good balance between image quality and computational efficiency.

[0094] Figure 8 A qualitative comparison is shown between the method described in this paper (image 802) and baseline methods UNet (801) and AHDR (803). From Figure 8As can be seen, in this example, UNet produces artifacts in faces and arms, while AHDR produces artifacts in cars and buildings. HDR images generated using the method described in this paper do not have these artifacts and provide better quality results. The method achieves optimal performance and is computationally efficient, maintaining a good balance between accuracy and efficiency.

[0095] The applicant hereby discloses individually each individual feature described herein, as well as any combination of two or more such features. With ordinary knowledge of those skilled in the art, such features or combinations can be implemented as a whole based on this specification, regardless of whether such features or combinations of features solve any of the problems disclosed herein, and without limitation to the scope of the claims. This application demonstrates that aspects of the invention can be constituted by any such individual features or combinations of features. Various modifications that can be made within the scope of the invention, given the foregoing description, will be apparent to those skilled in the art.

Claims

1. An image processing apparatus (700) comprising one or more processors (704), characterized in that, Used for: Receive (601) multiple input images (301, 302, 303); For each input image, a decomposed dataset (602) is formed by decomposing the input image (301, 302, 303) or its filtered version (307, 308, 309) into a first plurality of frequency-specific component sets (313), each frequency-specific component set representing the occurrence of a feature at a corresponding frequency interval in the input image or its filtered version; a convolution operation is performed on the low-frequency components in the first plurality of frequency-specific component sets (313) to form convolutional data (318, 319, 320); the convolutional data (318, 319, 320) is decomposed into a second plurality of frequency-specific component sets (324), each frequency-specific component set representing the occurrence of a feature at a corresponding frequency interval in the convolutional data; One or more convolutional neural networks are used to process (603) the high-frequency components in the second plurality of frequency-specific component sets (324) and the first plurality of frequency-specific component sets (313) to form a combined image dataset (327). A (604) construction operation is performed on the combined image dataset (327), the construction operation being adapted to construct images from multiple frequency-specific components to form an output image (333) representing the combination of the input images.

2. The image processing apparatus according to claim 1, characterized in that, The step of decomposing the input image includes performing a discrete wavelet transform operation on the input image.

3. The image processing apparatus according to claim 1 or 2, characterized in that, The construction operation is an inverse discrete wavelet transform operation.

4. The image processing apparatus according to claim 1 or 2, characterized in that, The device includes a camera (701) and is configured to respond to input from a user of the device, such that the camera captures the plurality of input images (301, 302, 303), each of which is captured with an exposure different from the other input images.

5. The image processing apparatus according to claim 1 or 2, characterized in that, The decomposed data is formed by decomposing the corresponding input image version filtered by the convolutional filter.

6. The image processing apparatus according to claim 1 or 2, characterized in that, The device is used for: Masking and weighting are applied to at least some regions of a decomposed dataset to form attention-filtered decomposed data. Select a subset of components of the attention-filtered decomposed data, wherein the components correspond to frequencies that are lower than the other components of the attention-filtered decomposed data; At least the components in the aforementioned subset are merged to form merged data. The merged data constitutes the input to the construction operation.

7. The image processing apparatus according to claim 6, characterized in that, The device is used to: decompose the attention-filtered data; and merge the relative low-frequency components of the attention-filtered data through multiple residual operations to form convolutional low-frequency data. A reconstruction operation is performed based on the relatively high-frequency components of the attention-filtered data and the low-frequency data after convolution.

8. The image processing apparatus according to claim 1, characterized in that, The device is used for: Merge a first subset of the second plurality of frequency-specific component sets to form first merged data; A mask and weighted combination are performed on a first subset of the second plurality of frequency-specific component sets and the first merged data to form a first combined data; Perform a first convolutional combination on a second subset of the second plurality of frequency-specific component sets to form a second combined data; The first combined data and the second combined data are upsampled to form the first upsampled data; A mask and weighted combination are performed on a first subset of the first plurality of frequency-specific component sets and the first upsampled data to form a third combined data; A second convolutional combination is performed on a second subset of the first plurality of frequency-specific component sets to form a fourth combined data; The third and fourth combined data are upsampled to form the second upsampled data; The output image is formed based on the second upsampled data.

9. The image processing apparatus according to claim 8, characterized in that, The first subset (409) is a subset of the relatively low-frequency components.

10. The image processing apparatus according to claim 8, characterized in that, The second subset (408) is a subset of the relatively high-frequency components.

11. The image processing apparatus according to claim 8, characterized in that, The output image is formed based on a combination of the second upsampled data and a convolutional version of the input image.

12. A computer-implemented image processing method (600), characterized in that, include: Receive (601) multiple input images (301, 302, 303); For each input image, a decomposed dataset (602) is formed by decomposing the input image (301, 302, 303) or its filtered version (307, 308, 309) into a first plurality of frequency-specific component sets (313), each frequency-specific component set representing the occurrence of a feature at a corresponding frequency interval in the input image or its filtered version; a convolution operation is performed on the low-frequency components in the first plurality of frequency-specific component sets (313) to form convolutional data (318, 319, 320); the convolutional data (318, 319, 320) is decomposed into a second plurality of frequency-specific component sets (324), each frequency-specific component set representing the occurrence of a feature at a corresponding frequency interval in the convolutional data; One or more convolutional neural networks are used to process (603) the high-frequency components in the second plurality of frequency-specific component sets (324) and the first plurality of frequency-specific component sets (313) to form a combined image dataset (327). A (604) construction operation is performed on the combined image dataset (327), the construction operation being adapted to construct images from multiple frequency-specific components to form an output image (333) representing the combination of the input images.

Images