Permutation invariant high dynamic range imaging

Abstract

Described herein is an image processing apparatus (1200) for forming an enhanced image, the apparatus comprising one or more processors (1204) configured to: receive (1101) one or more input images (301, 302, 404, 405, 406, 501, 502, 503, 601, 602, 603); form (1102) a respective feature representation from each of the one or more input images, each feature representation representing features of the respective input image; perform (1103) a symmetric pooling operation (305) on the one or more feature representations to form an enhanced image (408, 505, 608) from at least some of the features of the one or more feature representations identified by the symmetric pooling operation. The apparatus can generate images with greater dynamic range of sensitivity, greater bit depth and signal-to-noise ratio, and with less quantization error and richer color representation.

Description

TECHNICAL FIELD

[0001] The present application relates to estimating a High Dynamic Range (HDR) image from one or more Low Dynamic Range (LDR) images. BACKGROUND

[0002] The dynamic range of an image is defined as the ratio between the maximum and minimum measurable light intensities (white and black, respectively). The human visual system has a very wide dynamic range. Humans can perceive details in very bright and very dark scenes. However, due to the nature of the camera's photosensors, it is very difficult for a camera to capture such a wealth of details.

[0003] As shown in Figures 1(a) and 1(b), a camera's sensor is composed of photosites that act as light cavities that produce an electrical signal when a photon "falls" into the cavity. Due to the inherent noise level in the signal, the minimum number of photons that each cavity can detect, thus, sets the "minimum" light that the camera can measure. Figure 1(a) shows the minimum measurable light due to the sensor noise level. At the other end of the brightness, when there are too many electrons falling into the photosite, they can overflow and saturate, effectively setting the maximum measurable light, i.e. white. Figure 1(b) shows the maximum measurable light due to the photosite overflow. All the brightness in the scene that falls outside this range will be under- or over-exposed, thus, losing part of the scene brightness.

[0004] As shown in Figures 2(a) and 2(b), if the scene's brightness is high, several frames with different exposure values can help to show all the details of the scene. Figure 2(a) shows bright details around the door and window, but the back of the room is under-exposed. In Figure 2(b), the area around the door and window is over-exposed, but the details in the back of the room are well-exposed.

[0005] Current cameras have a dynamic range that is far from being able to capture all the details in high-contrast scenes. Therefore, in the case of a given static camera and scene, several frames are captured using different exposure values (i.e. auto-bracketing exposure) and merged by weighted fusion. This makes the resulting image able to preserve all the details in the input images, effectively increasing the dynamic range, which is called an HDR image.

[0006] Existing methods can generally be divided into two main areas: multi-frame fusion (combining multiple images with different exposures by linear weighting or neural networks) and single-image neural networks.

[0007] In multi-frame approaches, by surrounding the exposure images, different ranges can be captured and fused, thus exploiting a large amount of scene luminance information beyond the dynamic range of a single capture. However, multi-frame HDR brings several challenges. It requires precise alignment and pixel registration between different frames, which is very difficult, especially for high-speed motion scenes. Moreover, for certain types of photography, it can not be possible to take multiple frames because the relevant elements to be captured can only be available for a very short time (e.g. in wildlife photography or sports).

[0008] In contrast, single-image HDR processing does not require any alignment between frames and thus is able to efficiently reconstruct images with arbitrary amounts of motion. However, single-image HDR methods rely on hallucinated details in under- or over-exposed areas of low dynamic range input images. Therefore, they do not build well scenes with very wide luminance ranges and such techniques are best suited to reconstruct HDR images with only slightly higher luminance range than a single image.

[0009] There is a need to develop a method that overcomes these problems. SUMMARY

[0010] According to an aspect, the application provides an image processing apparatus for forming an enhanced image, the apparatus comprising one or more processors configured to: receive one or more input images; form a respective feature representation from each of the one or more input images, each feature representation representing features of the respective input image; perform a symmetric pooling operation on the one or more feature representations to form an enhanced image from at least some of the features of the one or more feature representations identified by the symmetric pooling operation.

[0011] The apparatus can process any number of input frames, process the frames in any order and process frames with any exposure values. Under- or over-exposed content can be hallucinated when it is not available in any of the input frames. The apparatus and method can process frames from different camera sensors / ISPs and process different levels of noise per frame. The apparatus can generate images with greater photographic dynamic range (i.e. the ability to capture both shadows and highlights within the same scene), greater bit depth and signal-to-noise ratio and with less quantization error and richer color representation.

[0012] The symmetric pooling operation can cause the or each feature representation to follow a common processing logic to identify components of the respective feature representation. Thereby components of each feature representation can be identified and used to form the enhanced image.

[0013] The symmetric pooling operation can operate in a way that is not affected by the order in which the feature representations are provided to the symmetric pooling operation. Therefore, the apparatus can process input frames in any order and has permutation invariance.

[0014] The symmetric pooling operation may include max pooling and / or average pooling. These may be convenient implementations for performing the symmetric pooling operation.

[0015] The one or more processors can be used to: perform a first convolution operation on each input image or data derived therefrom to form the corresponding feature representation; and perform a second convolution operation on the output of the symmetric pooling operation to form the enhanced image. Therefore, multiple convolutions can be applied to the input images to form the feature representation, then the symmetric pooling operation, which creates a “single feature representation” for all the input images (or a single image), and one or more convolutions can be applied to obtain the estimated HDR image.

[0016] The enhanced image has a higher dynamic range than each of the one or more input images. Thus, an HDR image can be formed from one or more LDR input images.

[0017] The one or more processors can be used to: perform a concatenation operation on the output of the symmetric pooling operation and the corresponding feature representation to form a corresponding intermediate feature representation. The second convolution operation can be performed on the intermediate feature representation. In a preferred implementation, this can be a single convolution operation after the symmetric pooling operation and a single convolution operation before the symmetric pooling operation. However, any number of convolutions can be used before and after the symmetric pooling operation.

[0018] The one or more processors can be used to form the representation of the feature by performing one or more of the following correction tasks: denoising, inverse quantization, linearization, camera response calibration, and overexposure and underexposure correction on the corresponding input image. The correction tasks can be intermediate correction tasks. This allows the deep learning model to include multiple neural networks operating sequentially in single-image and multi-image settings to perform dynamic range enhancement of low dynamic range images.

[0019] The symmetric pooling operation can be performed in conjunction with a trained neural network. For example, the convolution around the symmetric pooling block can be known and can be implemented as a neural network.

[0020] The image processing apparatus may include a camera (or other imaging device). Each processor may be used to receive the input image from the camera. The camera or other imaging device may be used to capture the input image at different times with different exposure settings. This allows the apparatus to process a series of LDR images captured at different exposures.

[0021] Exposure normalization can be performed on the at least one input image. This may improve image quality.

[0022] The apparatus can be used to receive a single input image. Alternatively, the one or more processors can be used to: receive multiple input images; form a corresponding feature representation from each of the multiple input images, each feature representation representing a feature of the corresponding input image; and perform a symmetric pooling operation on the feature representations to form an enhanced image by fusing at least some of the features of the one or more feature representations identified by the symmetric pooling operation. Therefore, the apparatus can process one or more LDR images to form the enhanced image.

[0023] The enhanced image can be an estimated high dynamic range image, and the dynamic range of the one or more input images is smaller than that of the high dynamic range image. Thus, an HDR image can be generated from one or more LDR input images.

[0024] According to a second aspect, the present invention provides a computer-implemented method for forming an enhanced image, the method comprising: receiving one or more input images; forming a corresponding feature representation from each of the one or more input images, each feature representation representing a feature of the corresponding input image; and performing a symmetric pooling operation on the one or more feature representations to form an enhanced image by at least some of the features of the feature representations identified by the symmetric pooling operation.

[0025] The method can process any number of input frames, in any order, and with any exposure value. Overexposed or underexposed content that is unavailable in any input frame can be phantomized. The method can process frames from different camera sensors / ISPs and handle different levels of noise for each frame.

[0026] The receiving step may include receiving multiple input images, each with the same field of view. Therefore, the input images may have the same field of view (e.g., from ordinary photographs).

[0027] The receiving step may include receiving multiple input images, and at least two of the input images have different fields of view. For example, the input images may be photographs captured sequentially. Attached Figure Description

[0028] The invention will now be described by way of example with reference to the accompanying drawings.

[0029] In the attached diagram:

[0030] Figures 1(a) and 1(b) schematically illustrate the dynamic range limitation caused by the sensor's photosensitive point. Figure 1(a) shows the minimum measurable light due to the sensor's noise level, and Figure 1(b) shows the maximum measurable light due to photosensitive point overflow.

[0031] Figures 2(a) and 2(b) show examples of photographs of the same scene with different exposure values;

[0032] Figure 3 A schematic diagram of a locally symmetric pooling block is shown;

[0033] Figures 4(a) and 4(b) illustrate examples of methods using global symmetric pooling. Figure 4(a) shows a single-image backbone where the input image is fed into the network via skip connections. In Figure 4(b), multiple input images are fed into this type of network separately.

[0034] Figure 5 Examples of methods using local and global symmetric pooling blocks are shown;

[0035] Figure 6 Schematic diagrams of single-image and multi-image HDR deep learning models that include different image processing operations are shown;

[0036] Figure 7 A schematic diagram illustrating an example of a network design for denoising / JDD and inverse quantization is shown;

[0037] Figure 8 A schematic diagram illustrating an example of a network architecture used for linearization is shown;

[0038] Figure 9 A schematic diagram illustrating an example of a network architecture used for exposure correction is shown;

[0039] Figure 10 A schematic diagram illustrating an example of a network architecture used for correction is shown;

[0040] Figure 11 An example of a computer-implemented method for forming an enhanced image is shown;

[0041] Figure 12 An example of a device for implementing the methods described herein is shown;

[0042] Figures 13(a) and 13(b) show examples of results obtained using the methods described herein. Detailed Implementation

[0043] Embodiments of the present invention relate to estimating a high dynamic range image based on one or more low dynamic range images.

[0044] This invention addresses the problem of obtaining HDR images from a single LDR image or a set of such images (i.e., multiple frames) through a modular deep learning model with invariant permutation. In some embodiments, the model can jointly perform denoising, inverse quantization, and reconstruction of underexposed and overexposed regions.

[0045] In summary, the image processing apparatus forms an enhanced image, i.e., an estimated HDR image. The apparatus is configured to receive one or more input images. The apparatus is configured to form a corresponding feature representation from each of the one or more input images, each feature representation representing a feature of the corresponding input image. The apparatus is configured to perform a symmetric pooling operation on the one or more feature representations to form the enhanced image using at least some of the features identified by the symmetric pooling operation from the one or more feature representations.

[0046] The symmetric pooling operation causes the feature representations to follow a common processing logic to identify the components of the respective feature representations. The feature representations are formed by applying a transformation to the input image. The feature representations summarize the features of the input image that are useful for the final task. In the present case, the feature representations identify features that are helpful for the HDR task and are compatible with the symmetric pooling operation.

[0047] The apparatus and method described herein can process any number of input frames (single or multiple, with any exposure value configuration and any order of input), and can process the fusion of image details and the phantom construction of missing details even when certain areas of the input image are unavailable.

[0048] The device achieves this flexibility through an arrangement-invariant perceptual design that leverages the advantage of using symmetric operations (e.g., max pooling) across different neural networks. Therefore, the operation of the symmetric pooling operation is unaffected by the order in which feature representations are provided to the symmetric pooling operation.

[0049] The algorithm used by the device can adapt any depth single-image HDR model to depth-permutation-invariant single-image and multi-image HDR models by including specific intermediate and global symmetric pooling operations in multiple identical copies of the model. This allows for the use of any number of inputs in any permutation with any exposure values.

[0050] In some embodiments described herein, the deep learning model can jointly perform denoising or joint denoising and demosaicing (JDD), inverse quantization, camera response calibration, and overexposure and underexposure correction as a series of building blocks for dynamic range enhancement. The method allows the use of images from different camera sensors / ISPs and is capable of handling different levels of noise per frame (i.e., each input image can have different levels of noise).

[0051] Embodiments of the present invention can provide a training strategy in which subtasks are trained independently without requiring fully annotated data. This makes it easier to prepare training data.

[0052] The network accepts any number of low dynamic range (LDR) images as input from multiple different sensors and ISPs in any arrangement. When the network receives multiple input images, these images can have different noise levels and different exposure values. In summary, the input is an arbitrary number of LDR images, and the output is an estimated HDR image.

[0053] The symmetric pooling operation is a means of providing a scalable fusion mechanism that can process any number of images. When only one image is provided as input, the symmetric pooling operation adopts a single-image model; when multiple frames are available, the symmetric pooling operation provides an efficient mechanism for sharing information through pooling.

[0054] The symmetric pooling operation is defined in such a way that if applied to a single input, it will not modify that input: the output will be the same as the input (i.e., identity transformation). Max pooling of a single image feature representation will result in the same image feature representation.

[0055] In the case of multiple images, the apparatus can be used to: receive multiple input images; form a corresponding feature representation from each of the multiple input images; and perform a symmetric pooling operation on the feature representations to form an enhanced image by fusing at least some of the features of the one or more feature representations identified by the symmetric pooling operation. The fused features may correspond to the same features of the corresponding images. Therefore, one or more features of each feature representation may at least partially (and in some cases completely) refer to the same image features.

[0056] For a single image, the input is Im_1. A feature extractor is applied to the image, i.e., F(Im_1). A max-pooling operation, max(F(Im_1)), is performed on this set of features. Max-pooling for a single image is only for identity, therefore max(Im_1) = Im_1, and the model works without requiring architectural changes.

[0057] This global max pooling can occur in any intermediate task described in the examples below (e.g., denoising, inverse quantization, overexposure and underexposure correction). Preferably, the global max pooling is applied at the end of the pipeline, such as the overexposure and underexposure correction stage.

[0058] For permutation invariance, preferably, the global symmetric pooling is performed at least once, and then all heads are collapsed into a single head after the global symmetric pooling. In contrast, local symmetric pooling can be performed at the desired frequency and does not collapse the heads into a single head.

[0059] For the two input images 301 and 302, Figure 3 The mechanism of the local symmetric pooling operation is illustrated.

[0060] For each input image 301, 302, a first convolution operation 303, 304 is performed to form the corresponding feature representation. The symmetric pooling operation (max pooling in this example) is shown at 305. The outputs are concatenated at 306, 307, and the enhanced image is formed by performing a second convolution operation 308, 309 on the outputs. The concatenated operations 306, 307 form corresponding intermediate feature representations, and the second convolution operation 308, 309 is performed on these intermediate feature representations.

[0061] Figures 4(a) and 4(b) illustrate a method using only global symmetric pooling operations. Figure 4(a) shows a single-image backbone where input image 401 is fed into network 402 via skip connections to generate input 403. In Figure 4(b), multiple input images 404, 405, and 406 are fed into such a network. Multiple encoders share weights, with one weight for each input image. Symmetric pooling layers (max pooling in this example) are applied to multiple copies of the network in the generated feature representation, as shown at 407. The enhanced output image is shown at 408.

[0062] Figure 5 A method using local and global symmetric pooling is illustrated in the encoder 504. A first convolution operation is performed on each input image 501, 502, 503 to form the corresponding feature representation. The outputs of the local symmetric pooling operations are concatenated, and the HDR image is formed by performing a second convolution operation on the outputs. The concatenation operation forms corresponding intermediate feature representations, and the second convolution operation is performed on the intermediate feature representations. Local symmetric pooling enables the sharing of more information channels among different copies of the network or head, while each head retains salient features due to the concatenation operation. The HDR output image is shown at 505.

[0063] Figure 6 Exemplary summaries of various levels of single-image and multi-image HDR deep learning models are shown. Figure 6 The input image is shown at positions 601, 602, and 603. In this example, the deep learning model includes four neural networks that operate sequentially in single-image and multi-image settings to perform dynamic range enhancement of low dynamic range images. Figure 6 In the diagram, the architecture is in the following stages: denoising / JDD and inverse quantization 604, linearization 605, overexposure correction 606, and correction 607. The generated estimated HDR image is shown at 608.

[0064] Figure 7 The denoising architecture is illustrated. The network performs denoising / JDD and inverse quantization collectively. It has a U-Net structure with skip connections between the encoder and decoder, as indicated by the arrows. The noisy input image is shown at 701. The feature representation formed from the input image is passed through convolutional layer 702. Symmetric pooling layers are applied in multiple copies of the network (one for each input image) with shared weights. The downsampling and upsampling are shown at 703 and 704, respectively. The upsampled feature map is then passed through convolutional layer 705 to provide a denoised output at 706.

[0065] Figure 8 The network architecture shown performs linearization by predicting the weights needed to generate the inverse camera response function, which will be applied to the nonlinear input image 801. The network has a ResNet structure including residual blocks. The input 801, along with Sobel filter features 802 and histogram 803, is passed through convolutional layers and residual blocks, as shown at 804. Pooling is performed at 805, the inverse camera response function (CRF) step is performed at 806, and the CRF is executed again at 807 to provide a linearized output at 808.

[0066] Figure 9 The network architecture shown performs exposure correction by phantomizing details in the overexposed regions of the image and fusing details from multiple sources when there are multiple input images. It has a U-net structure with skip connections between the encoder and decoder, as indicated by the arrows. The overexposed input is shown at 901. After passing through a convolutional layer at 903, the input is downsampled, as shown at 902. At 904, upsampling is performed by applying symmetric pooling layers in multiple copies of the network with shared weights (one for each input image). After passing through a convolutional layer at 905 and cascading at 906, the corrected input is shown at 907.

[0067] Figure 10 The network architecture shown performs corrections on the image to improve visual quality. It has a U-Net structure with skip connections between the encoder and decoder. The network takes the denoising output 706, the exposure correction output 907, and the linearization output 808, passes them through the convolutional layer 1001 before downsampling 1002 and upsampling 1003, and then through another convolutional layer 1004 before cascading at 1005 to provide the corrected output 1006.

[0068] In the preferred implementation, such as Figure 6 As shown, the deep learning model sequentially includes combining Figures 7 to 10 The four networks described. For simplicity, Figures 7 to 10 A single copy of each model is shown, but it is best to use multiple copies of the same model with shared weights for denoising and exposure correction, such as... Figure 6 As shown.

[0069] To train the network, each subtask is trained separately first. To train the denoising / JDD network, the predicted image D is minimized to the greatest extent possible. Pred and ground truth image D GT The average of the squared differences between pixels. If N is the number of pixels in the image, then the loss function is:

[0070]

[0071] To train the linearized network, the predicted image L is minimized to the greatest extent possible. Pred and ground truth image L GT The average of the squared differences between pixels. Furthermore, the predicted inverse CRF is minimized to the greatest extent possible. Pred Inverse CRFC of ground truth GT The squared difference between them. If N is the number of pixels in the image and M is the number of elements in the predicted inverse CRF, then the loss is derived by the following equation:

[0072]

[0073] To train the exposure correction network, the μ-Law tone mapping function and γ = 10 are used to adjust the predicted image H. Pred Perform tone mapping to obtain H Pred_tonemapped The ground truth image H GT The same method is used for tone mapping to obtain H. GT_tonemapped Calculate the overexposed region H of the input image. mask_overexposed The mask and the non-overexposed area 1-H maskoverexposedThe masks are applied to the tone-mapped predicted and ground truth images to obtain H. Pred_overexposed H Pred_normal H GT_overexposed H GT_normal .

[0074] Then, the loss function is derived using the pixel-by-pixel reconstruction loss (L1 loss) on the masked image, as shown below:

[0075]

[0076] After training the three subtasks individually, the three networks can be trained together to improve performance:

[0077] L Finetune =L Exposure +L Linearize +L Denoise

[0078] To train the corrected network, this network and the other three subnetworks are jointly fine-tuned. The goal is to minimize the predicted image R after tone mapping. Pred_tonemapped and the ground truth image R after tone mapping GT_tonemapped Pixel-wise reconstruction loss (L1 loss):

[0079]

[0080] For a multi-image HDR dataset, the single-image HDR reconstruction dataset can be used to train the model. A series of transformations are applied to the ground truth HDR images to generate LDR-HDR pairs for training.

[0081] In one example, the dimensions of the longer and shorter sides of the HDR image are adjusted to 1024 and 512, respectively. The image is then split into two 512x512 ground truth HDR images. This set of images represents the ground truth for the exposure correction and remediation network. A burst size b, between 1 and n, is then randomly selected. b exposure values ​​within the range [-8, 8] are randomly sampled and applied to the ground truth images to obtain b exposure images. The dynamic range of these images is cropped to between 0 and 1. This represents the ground truth for the linearization network. A CRF is randomly sampled from the Database of Response Functions (DoRF) and applied to the cropped images. This represents the ground truth for the denoising network. Noise is added to two components: a fixed element and a signal-dependent element. Using the noise model, the variance of the ground truth is approximated using the noisy images, and the estimated variance is concatenated to each image. All images within the burst range are aligned to the exposure values.

[0082] Therefore, embodiments of the present invention can provide apparatus for implementing an algorithm that, by incorporating the specific intermediate and global symmetric pooling operations described herein (e.g., average or max pooling), enables any single-image HDR model to become permutation-invariant single-image and multi-image HDR models. The deep learning models described herein can jointly perform denoising / JDD, inverse quantization, camera response calibration, and oversaturation and undersaturation correction as continuous building blocks for single-frame and multi-frame dynamic range enhancement. Embodiments of the present invention can provide a training strategy by which subtasks are trained individually without requiring fully annotated data.

[0083] Figure 11 An example of a computer-implemented image processing method is shown. In step 1101, the method includes receiving one or more input images. In step 1102, the method includes forming a corresponding feature representation from each of the one or more input images, each feature representation representing a feature of the corresponding input image. In step 1103, the method includes performing a symmetric pooling operation on the one or more feature representations to form an enhanced image from at least some of the features of the feature representations identified by the symmetric pooling operation.

[0084] The apparatus may include an imaging device, such as a camera. When using multiple images as input, the apparatus may be used to respond to input from a user of the apparatus, causing the camera or other imaging device to capture each input image at a different exposure than the other input images. The captured images may also have different noise levels.

[0085] Figure 12An example of an apparatus 1200 including an imaging device 1201 (e.g., a camera) is shown, the apparatus being used to process image data captured by at least one image sensor in the apparatus using the methods described herein. The apparatus 1201 includes image sensors 1202, 1203. Such an apparatus 1201 includes some onboard processing capabilities. This may be provided by the processor 1204. The processor 1204 may also be used for the basic functions of the apparatus.

[0086] Transceiver 1205 is capable of communicating with other entities 1210, 1211 via a network. These entities may be physically located away from device 1201. The network may be a publicly accessible network, such as the Internet. Entities 1210, 1211 may be cloud-based. Entity 1210 is a computing entity. Entity 1211 is a command and control entity. These entities are logical entities. In practice, each of them may be provided by one or more physical devices (e.g., servers and data storage areas), and the functionality of two or more of the entities may be provided by a single physical device. Each physical device implementing an entity includes a processor and memory. The device also includes a transceiver for sending and receiving data to and from transceiver 1205 of device 1201. The memory stores code in a non-transitory manner, which can be executed by the processor to implement the corresponding entity in the manner described herein.

[0087] The command and control entity 1211 can train the model used in the device. This is typically a computationally intensive task, even though the obtained model can be efficiently described. Therefore, the development of the algorithm can be efficiently performed in the cloud, where a large amount of energy and computing resources are expected to be available. This is expected to be more efficient than forming such a model on a typical imaging device.

[0088] In one implementation, after the algorithm is developed in the cloud, the command and control entities can automatically generate a corresponding model and transmit it to the relevant imaging device. In this example, the model is implemented by processor 1204 at device 1201.

[0089] In another possible implementation, images can be captured by one or both of the sensors 1202 and 1203, and the transceiver 1205 can send the image data to the cloud for processing. The generated image can then be sent back to the device 1201, such as... Figure 12 As shown in 1212.

[0090] Therefore, the method can be deployed in various ways, such as in the cloud, on the device, or in dedicated hardware. As mentioned above, cloud facilities can perform training to develop new algorithms or improve existing ones. Depending on the computing power near the data corpus, the training can be performed close to the source data or in the cloud, for example, using an inference engine. The method can also be implemented on the device, in dedicated hardware, or in the cloud.

[0091] Figures 13(a) and 13(b) show examples of results obtained using the method described herein (using a single-image HDR test set). Figure 13(a) shows the results using a single image, and Figure 13(b) shows examples of results using multiple images (two images in this example). In both cases, an improvement in image quality relative to the input was observed.

[0092] The apparatus and method described herein can process any number of input frames, in any order, and with any exposure value. Overexposed or underexposed content can be phantomized when it is unavailable in any input frame. The apparatus and method can process frames from different camera sensors / ISPs and handle different levels of noise for each frame. The deep learning model can be trained without fully annotated data (only paired tasks are required).

[0093] This deep learning pipeline retains the advantages of both single-frame and multi-frame HDR models.

[0094] The apparatus and methods described herein can generate images with a greater dynamic range (i.e., the ability to capture shadows and highlights within the same scene), greater bit depth and signal-to-noise ratio, as well as smaller quantization errors and richer color representations (e.g., 16-bit RGB images per channel).

[0095] The applicant hereby discloses individually each individual feature described herein, as well as any combination of two or more such features. With ordinary knowledge of those skilled in the art, such features or combinations can be implemented as a whole based on this specification, regardless of whether such features or combinations of features solve any of the problems disclosed herein, and without limitation to the scope of the claims. This application demonstrates that aspects of the invention can be constituted by any such individual features or combinations of features. Various modifications that can be made within the scope of the invention, given the foregoing description, will be apparent to those skilled in the art.

Claims

1. An image processing apparatus for forming an enhanced image, characterized in that, The device includes one or more processors, which are used to learn modular deep learning models with invariant permutations: Receive one or more input images; Each of the one or more input images forms a corresponding feature representation, and each feature representation represents a feature of the corresponding input image. A symmetric pooling operation is performed on the one or more feature representations, the manner in which the symmetric pooling operation is performed is not affected by the order in which the feature representations are provided to the symmetric pooling operation; An enhanced image is formed using at least some of the features represented by the one or more features identified by the symmetric pooling operation.

2. The image processing apparatus according to claim 1, characterized in that, The symmetric pooling operation causes each feature representation to follow a common processing logic to identify the components of the corresponding feature representation.

3. The image processing apparatus according to claim 1 or 2, characterized in that, The symmetric pooling operation includes max pooling and / or average pooling.

4. The image processing apparatus according to claim 1 or 2, characterized in that, The one or more processors are used for: Perform a first convolution operation on each input image or the data derived therefrom to form the corresponding feature representation; The enhanced image is formed by performing a second convolution operation on the output of the symmetric pooling operation.

5. The image processing apparatus according to claim 1 or 2, characterized in that, The enhanced image has a higher dynamic range than each of the one or more input images.

6. The image processing apparatus according to claim 4, characterized in that, The one or more processors are used for: The output of the symmetric pooling operation and the corresponding feature representation are concatenated to form the corresponding intermediate feature representation; The second convolution operation is performed on the intermediate feature representation.

7. The image processing apparatus according to claim 1 or 2, characterized in that, The one or more processors are configured to form each feature representation by performing one or more of the following correction tasks: denoising, inverse quantization, linearization, camera response calibration, and overexposure and underexposure correction on the corresponding input image.

8. The image processing apparatus according to claim 1 or 2, characterized in that, The symmetric pooling operation is performed in conjunction with a trained neural network.

9. The image processing apparatus according to claim 1, characterized in that, Includes a camera, wherein the one or more processors are used to receive the input image from the camera.

10. The image processing apparatus according to claim 9, characterized in that, The camera is used to capture one or more input images at different times with different exposure settings.

11. The image processing apparatus according to claim 1 or 2, characterized in that, Exposure normalization processing is performed on the one or more input images.

12. The image processing apparatus according to claim 1 or 2, characterized in that, The device is used to receive a single input image.

13. The image processing apparatus according to claim 1 or 2, characterized in that, The one or more processors are used for: Receive multiple input images; Each of the plurality of input images forms a corresponding feature representation, and each feature representation represents a feature of the corresponding input image. A symmetric pooling operation is performed on the feature representations to form an enhanced image by fusing at least some of the features identified by the symmetric pooling operation on one or more feature representations.

14. The image processing apparatus according to claim 1 or 2, characterized in that, The enhanced image is an estimated high dynamic range image, and the dynamic range of the one or more input images is smaller than that of the high dynamic range image.

15. A computer-implemented method for forming an enhanced image, characterized in that, Applied to an image processing apparatus, the apparatus including one or more processors, the processors including modular deep learning models with invariant permutations, the method includes: Receive one or more input images; Each of the one or more input images forms a corresponding feature representation, and each feature representation represents a feature of the corresponding input image. A symmetric pooling operation is performed on the one or more feature representations, the manner in which the symmetric pooling operation is performed is not affected by the order in which the feature representations are provided to the symmetric pooling operation; An enhanced image is formed using at least some of the features represented by the features identified by the symmetric pooling operation.

16. The method according to claim 15, characterized in that, The receiving step includes receiving multiple input images, and each input image has the same field of view.

17. The method according to claim 15, characterized in that, The receiving step includes receiving multiple input images, and at least two of the input images have different fields of view.

Images