High Dynamic Range (HDR) Photography Using Partitioned Compute on Wearable Devices

Abstract

1. A method comprising: capturing, by a wearable device, a plurality of images each having a first resolution; processing, by the wearable device, the plurality of images to generate a first image having a second resolution, the second resolution being smaller than the first resolution; selecting, by the wearable device, a second image from the plurality of images having the first resolution based on a setting of the wearable device; and communicating, by the wearable device, the first image and the second image to a companion device; and processing, by the companion device, the first image; and merging, by the companion device, the processed first image with the second image to generate a high dynamic range (HDR) image.

Description

[Technical Field]

[0001] The embodiments relate to a digital photography process for images captured using a wearable device. [Background technology]

[0002] High dynamic range (HDR) photography is the standard for flagship mobile phones. The goal is to design a pipeline that takes as input a burst of images (e.g., about 10, possibly underexposed frames). The image burst is then aligned, merged, and denoised to produce a single HDR image. Tone mapping of the HDR image can be performed, and / or post-processing of other images can be performed. HDR is then made accessible to mobile device users. Summary of the Invention

[0003] The wearable device may be communicatively coupled to a companion device to divide computing operations. In an exemplary implementation, the wearable device may capture an image burst. The image burst may be used to generate a first image having a smaller resolution than the images in the image burst. The wearable device may select a second image from the image burst. The first image and the second image may be communicated to the companion device. The companion device may generate an HDR image by merging the first image and the second image.

[0004] In a general aspect, a device, system, non-transitory computer-readable medium (having computer-executable program code stored thereon) and / or method can perform a process using a method including capturing, by a wearable device, a plurality of images having a first number of pixels; processing, by the wearable device, the plurality of images to generate a first image having a second number of pixels that is less than the first number of pixels; selecting, based on a setting of the wearable device, one of the plurality of images having the first number of pixels as a second image; and communicating, by the wearable device, the first image and the second image to a companion device to generate a high dynamic range (HDR) image.

[0005] Implementations may include one or more of the following features, or any combination thereof. For example, the method may further include controlling, by the wearable device, the companion device to generate an HDR image based on the first image and the second image. Capturing the plurality of images may be an image burst including capturing a series of images in response to a trigger. Processing the plurality of images to generate the first image may include compressing at least one of the first image and each of the plurality of images. Processing the plurality of images to generate the first image may include downsampling at least one of the first image and each of the plurality of images.

[0006] The method may further include receiving a signal from the companion device, where capturing the plurality of images is responsive to receiving the signal. Processing the plurality of images to generate the first image may include reducing a number of pixels in each of the plurality of images to a second number of pixels, aligning corresponding pixels in the plurality of images, and merging the plurality of images to generate the first image. Processing the plurality of images to generate the first image may include tone mapping and color adjusting the images having the second number of pixels.

[0007] In another general aspect, a device, system, non-transitory computer-readable medium (having computer-executable program code stored thereon) and / or method can perform a process using the method, the method including receiving, by a companion device, from a wearable device, a first image having a first resolution and a second image having a second resolution, the first resolution being smaller than the second resolution, the first image and the second image being generated based on an image burst including capturing a series of images in response to a trigger, the method further including processing, by the companion device, the first image, and merging, by the companion device, the first image with the second image to generate a high dynamic range (HDR) image.

[0008] Implementations may include one or more of the following features, or any combination thereof. For example, merging the first image with the second image may include inputting the first image and the second image into a machine-learned model and generating an HDR image based on the first image together with the second image using the machine-learned model. The machine-learned model may be trained using HDR ground truth images. The method may further include controlling, by the companion device, the wearable device to initiate generation of the HDR image based on the first image and the second image. The method may further include communicating a signal to the wearable device, the signal causing the wearable device to trigger an image burst.

[0009] In yet another general aspect, a device, system, non-transitory computer-readable medium (having computer-executable program code stored thereon) and / or method can perform a process using a method, the method including capturing, by a wearable device, a plurality of images, each having a first resolution, and processing, by the wearable device, the plurality of images to generate a first image having a second resolution, the second resolution being smaller than the first resolution, the method further including selecting, by the wearable device, a second image from the plurality of images having the first resolution based on a setting of the wearable device, communicating, by the wearable device, the first image and the second image to a companion device, processing, by the companion device, the first image, and merging, by the companion device, the processed first image with the second image to generate a high dynamic range (HDR) image.

[0010] Implementations may include one or more of the following features, or any combination thereof. For example, processing the multiple images to generate the first image may include reducing the resolution of each of the multiple images to a second resolution, aligning corresponding pixels of the multiple images, and merging the multiple images to generate the first image. Processing the first image may include tone mapping and color adjusting the first image. For example, merging the first image with the second image may include inputting the processed first image and the second image into a machine-learned model and using the machine-learned model to generate an HDR image based on the processed first image together with the second image. The machine-learned model may be trained using HDR ground truth images.

[0011] The method may further include controlling the wearable device by the companion device to initiate generation of an HDR image based on the first image and the second image. The method may further include communicating a signal from the companion device to the wearable device that triggers the wearable device to capture a plurality of images. The method may further include controlling the companion device by the wearable device to generate the HDR image based on the first image and the second image. Capturing a plurality of images may be an image burst that includes capturing a series of images in response to a single trigger.

[0012] The exemplary embodiments will be more fully understood from the following detailed description herein and the accompanying drawings, in which like elements are indicated with like reference numerals and are given by way of example only, and therefore not by way of limitation of the exemplary embodiments. [Brief explanation of the drawings]

[0013] [Figure 1] 1 illustrates a system for generating HDR images, according to an exemplary embodiment. [Figure 2] 1 illustrates a system for generating HDR images, according to an exemplary embodiment. [Figure 3] 1 illustrates a method of operating a wearable device to generate HDR images, according to an exemplary embodiment. [Figure 4] 1 illustrates a method of operating a companion device to generate HDR images, according to an exemplary implementation. [Figure 5A] 1 illustrates a wearable device configured to generate HDR images, according to an exemplary implementation. [Figure 5B] 1 illustrates a companion device configured to generate HDR images, according to an exemplary implementation. [Figure 6] 1 illustrates a wearable device according to an exemplary embodiment. [Figure 7] 1 illustrates an example of a computing device and a mobile computing device in accordance with at least one exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] It should be noted that these figures are intended to illustrate general features of methods and / or structures utilized in certain exemplary embodiments and to supplement the written description provided below. However, these figures are not to scale, may not precisely reflect the precise structural or performance characteristics of any given embodiment, and should not be construed as defining or limiting the range of values ​​or characteristics encompassed by the exemplary embodiments. For example, the arrangement of modules and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in various figures is intended to indicate the presence of similar or identical elements or features.

[0015] Generating HDR images on a wearable device (e.g., smart glasses) that includes a camera can pose some challenges. For example, a wearable device may be computationally limited. In other words, using a wearable device can make the HDR processing pipeline take too long to execute. For example, a wearable device may be 10 times slower than the typical processing time on a mobile phone. Therefore, using the same processing pipeline typically used on a mobile phone on a wearable device may be undesirable from a user's perspective.

[0016] Furthermore, transferring image bursts from the wearable to a mobile phone capable of generating HDR images may take too long (e.g., more than 30 seconds) per image burst. Additionally, transferring image bursts from the wearable to the mobile phone may additionally use an excessive amount of resources, thereby reducing the battery charge of the wearable device. A reduced battery charge in the wearable device may render the wearable device unusable after capturing a few HDR images. Power constraints (e.g., design criteria) of the wearable device may include ensuring sufficient battery charge to enable image capture and processing on the wearable device for a predefined number of image captures (e.g., more than 200).

[0017] As described above, problems can arise when current HDR image processing techniques are used in wearable devices. In an exemplary implementation, these problems are solved using split compute in the wearable device process. Split compute in the wearable device process can include performing some of the image processing in the wearable device and some of the image processing in a companion device (e.g., a companion device communicatively coupled to the wearable device). For example, in an exemplary implementation, alignment and noise removal can be performed by the wearable device on an image having a first (e.g., lower) resolution (e.g., number of pixels) based on having a second (e.g., higher) resolution. Additional processing (e.g., tone mapping) can be performed on the companion device on the image having the first (e.g., lower) resolution. The image having the second (e.g., higher) resolution can then be merged (after processing) with the image having the first (e.g., lower) resolution to generate an HDR image.

[0018] FIG. 1 illustrates a system for generating an HDR image according to an exemplary embodiment. As shown in FIG. 1, the system includes a user 105, a wearable device 110, and a companion device 125. Also shown in FIG. 1 are a first image 115, a second image 120, and an HDR image 130. The wearable device 110 can be configured to generate the first image 115 and the second image 120. The wearable device 110 can be configured to communicate the first image 115 and the second image 120 to the companion device 125. The companion device 125 can be configured to generate the HDR image 130 based on the first image 115 and the second image 120.

[0019] The wearable device 110 may be, for example, a smart glasses device (e.g., an AR glasses device), a head-mounted display (HMD), an AR / VR device, a wearable computing device, etc. The user 105 views a real-world view in any direction. The wearable device 110 may be configured to generate an image burst (e.g., approximately 10, possibly underexposed frames) of the real world. The first image 115 and the second image 120 may be generated based on the image burst. For example, the wearable device 110 may capture the image burst as multiple images having a relatively high resolution (e.g., 12 megapixels). However, as described above, generating an HDR image from multiple images having a relatively high resolution on the wearable device 110 may be undesirable. Therefore, the wearable device 110 may be configured to compress (e.g., reduce the number of pixels) the multiple images having a relatively high resolution to a lower resolution (e.g., 3 megapixels). The multiple images having lower resolutions can then be aligned, merged, and / or denoised (image merging and denoising can often be one process) to generate the first image 115. The resolution (e.g., number of pixels) of the lower-resolution second image can be large enough to ensure that a visually (e.g., visually to a user) high-quality HDR image can be generated, but small enough to minimize resource (e.g., processing, memory, battery) usage. The second image 120 can be selected from multiple images having relatively high resolutions (the second image 120 can sometimes be referred to as a noisy image because the image has not been processed to remove noise inherent in the image sensor used to capture the image). The second image 120 can be selected based on settings of the wearable device. For example, preconfigured settings or user settings can be stored in memory.This setting may indicate, for example, that the first, last, middle, and / or other such image in a time sequence of multiple images (e.g., an image burst) having relatively high resolution should be selected.

[0020] The companion device 125 can be configured to receive the first image 115 and the second image 120 (e.g., via a wired and / or wireless connection). The first image 115 may be further processed by the companion device 125. For example, the first image may be tone mapped. The companion device 125 can be configured to generate an HDR image 130 based on the first image 115 (e.g., the further processed first image 115) and the second image 120. For example, a machine-learned model can be used to merge the first image 115 and the second image 120. The machine-learned model can be a deep learning architecture (e.g., a trained neural network, U-Net, etc.). The machine-learned model can be trained to generate an image (e.g., an HDR image) having a relatively high resolution (e.g., 12 megapixels) of the second image 120 and a post-processed (e.g., denoised) version of the first image 115. In other words, a machine-learned model can be trained to generate an HDR image based on the first image 115 and the second image 120.

[0021] 2 illustrates a system for generating an HDR image according to an exemplary embodiment. As shown in FIG. 2, the system includes a wearable device 110 and a companion device 125. The wearable device includes a plurality of images 205, an image processing module 210, a first image 215-1, and a second image 220-1. The companion device includes a first image 215-2, a second image 220-2, an image processing module 230, an image merging module 235, and an HDR image 240.

[0022] The wearable device 110 may include a camera (e.g., an image sensor) configured to capture high resolution (e.g., 12 megapixels or greater). The camera may be configured to capture images as an image burst. An image burst may include capturing a series of images in response to a single trigger (e.g., a button press, a shutter press, a command). An image burst may be multiple RAW images (e.g., unprocessed, uncompressed, low-quality, underexposed, noisy, and / or other images). The number of images in an image burst is variable and may be based on the amount of lighting in the environment, the dynamic range of the scene, and / or other factors. The image burst may be stored in memory (e.g., a cache) as multiple images 205. The wearable device 110 may be configured to control (e.g., communicate signals, commands, and / or data to) the companion device 125 to generate HDR images.

[0023] The image processing module 210 may be configured to generate a first image 215-1 and a second image 220-1 based on the plurality of images 205. Generating the first image 215-1 may include performing some image processing on the plurality of images 205. Generating the second image 220-1 may include selecting one of the plurality of images 205. The second image 220-1 may be selected based on a setting of the wearable device. For example, a preconfigured setting or a user setting may be stored in memory. The setting may indicate, for example, that the first, last, middle, and / or other such image in a time sequence of the plurality of images 205 should be selected.

[0024] The image processing module 210 can be configured to compress (e.g., reduce the number of pixels) the multiple images 205. The image processing module 210 may use an image resizing algorithm to compress the multiple images 205. The image resizing algorithm can proportionally reduce the number of pixels per row (width) and the number of pixels per column (height). For example, the width of each image can be reduced to n pixels, and the height can be proportionally reduced using an image resizing programming function such as resize (e.g., resize(n,0)). Other resizing algorithms, including machine learning models, are within the scope of this disclosure. In an exemplary implementation, the multiple images 205 can be compressed during alignment and merging operations. For example, the multiple images 205 can be downsampled during alignment and merging operations.

[0025] The image processing module 210 may be configured to align and merge multiple images 205. Aligning and merging multiple images 205 can remove noise from the multiple images 205. The multiple images 205 can be aligned using a Gaussian pyramid technique, progressing from coarse to finer alignment. Each level of the pyramid can be downsampled. Aligning the multiple images 205 can include aligning corresponding pixels in the multiple images 205 (e.g., moving pixels representing the same point so that they overlap each other). Aligning the multiple images 205 can prevent motion blur or ghosting. Merging the multiple images 205 can generate a single image that includes similar information from other aligned images, thereby reducing noise associated with the resulting image (e.g., first image 215-1) relative to the reference image. Merging can include using a variation of a Wiener filter on each tile (e.g., portion of an image) across the time dimension. The merging may include generating tiles (portions of images) for each of the multiple images and using a two-dimensional (2D) DFT over the corresponding tiles to result in merging of the images in the spatial frequency domain. The merging may improve color accuracy in dark areas of the resulting image (e.g., the first image 215-1).

[0026] Wearable device 110 and companion device 125 may be communicatively coupled using wired (e.g., Ethernet) and / or wireless (e.g., Bluetooth®) standards. As a result, a first image 215-1 and a second image 220-1 may be communicated from wearable device 110 to companion device 125. In FIG. 2 , first image 215-2 and second image 220-2 represent the same images as first image 215-1 and second image 220-1, respectively, except that first image 215-2 and second image 220-2 are stored in a memory (e.g., a cache) of companion device 125. Companion device 125 may be configured to generate an HDR image from an image burst captured by a camera (e.g., an image sensor) of companion device 125. Accordingly, example implementations may use elements of an HDR pipeline associated with companion device 125. Companion device 125 can be configured to control (e.g., communicate signals, commands, and / or data to) wearable device 110 to initiate capture of HDR images. For example, companion device 125 can be configured to control wearable device 110 to trigger capture of multiple images 205 (e.g., trigger an image burst). Accordingly, wearable device 110 can act as an input or peripheral device (e.g., image and / or camera input) to companion device 125.

[0027] The image processing module 230 may be configured to color map and / or tone map (or adjust) the first image 215-2. Color mapping and / or tone mapping may be an optional operation. Color mapping and / or tone mapping may be configured to map pixels to different color (e.g., RGB) values ​​based on the color and / or hue associated with the pixel and / or surrounding pixels. Color mapping and / or tone mapping may be configured to produce a more natural-looking image. Color mapping and / or tone mapping may be configured to reduce the dynamic range or contrast ratio of the entire image while preserving local contrast. The image processing module 230 may be configured to perform other image finishing operations. For example, the image processing module 230 may be configured to perform lens shading correction, white balancing, demosaicing, chromatic aberration correction, sharpening, and / or the like.

[0028] The image merge module 235 can be configured to generate an HDR image 240 based on the first image 215-2 and the second image 220-2. The image merge module 235 can be configured to merge the first image 215-2 with the second image 220-2 using a machine-learned model. The machine-learned model can be a deep learning architecture (e.g., U-Net, CNN, and / or the like). The machine-learned model can be trained to generate an image having a relatively high resolution (e.g., 12 megapixels) relative to the second image 220-2 and a post-processed (e.g., denoised) version of the first image 215-2. In other words, the machine-learned model can be trained to generate an HDR image based on the first image 215-2 and the second image 220-2.

[0029] The machine-learned model may be a machine-learned encoder-decoder architecture (e.g., U-Net, CNN, and / or the like) in which an input image is input to an encoder and an output image is output from a decoder. The difference between the input image and the output image may be based on training of the machine-learned encoder-decoder architecture. The image merging module 235 may be configured to generate the HDR image 240 based on the first image 215-2 and the second image 220-2. Thus, the machine-learned encoder-decoder architecture, in which the first image 215-2 and the second image 220-2 may be input to an encoder and the HDR image 240 may be output from the decoder, may merge the first image 215-2 and the second image 220-2 to generate the HDR image 240.

[0030] As described above, the first image 215-2 has a first resolution and the second image 220-2 has a second resolution, where the first resolution (e.g., 3 megapixels) is smaller than the second resolution (e.g., 12 megapixels). Thus, the encoder of the machine learning encoder-decoder architecture can be configured to receive two images of different sizes as input. The first element of the encoder can be a convolution element having a first number of channels as input and a second number of channels (often fewer than the first number) as output. The number of channels can be based on the number of pixels. For example, each pixel can have channels associated with the pixel's red, green, and blue colors (other representations of pixels are within the scope of this disclosure). In an exemplary implementation, the first element of the encoder (e.g., convolution) can include a sufficient number of input channels for the number of pixels associated with the first image 215-2 and the second image 220-2.

[0031] In an exemplary implementation, the encoder may include two encoders, and the outputs of the two encoders may be added together (e.g., pixel-by-pixel or channel-by-channel sum) as input to the decoder. The decoder may include a number of input channels sufficient for the additional channels of the two encoders. In an exemplary implementation, one of the two encoders may include a number of input channels sufficient for the number of pixels associated with the first image 215-2, and the other encoder may include a number of input channels sufficient for the number of pixels associated with the second image 220-2. As described above, the first image 215-2 has a first resolution and the second image 220-2 has a second resolution, where the first resolution (e.g., 3 megapixels) is smaller than the second resolution (e.g., 12 megapixels). Thus, the two encoders may have different numbers of input channels. Alternatively, the two encoders may have the same number of input channels, and the input of the encoder associated with the first image 215-2 may be configured to zero-pad input channels that do not have associated pixels input to the channel.

[0032] At the point where two encoder outputs are added together, the number of channels may be equal or unequal. After adding the outputs together, the result (e.g., the added channel) may be input directly to a decoder. Alternatively, the result may undergo additional encoder operations. For example, the result may undergo another convolution operation. The exemplary implementation is described as if the outputs of two encoders are added together. However, in the exemplary implementation, the element configured to add a channel may be in an intermediate step of one of the encoders (e.g., between the two convolutions). Thus, the output of an encoder (e.g., the encoder associated with the first image 215-2) may be input to another encoder (e.g., the encoder associated with the first image 215-2) that can add a channel.

[0033] Training the machine-learned model may include using HDR ground truth images. For example, training images representing a first image (e.g., a low-resolution, image-processed image) and a second image (e.g., a high-resolution, raw image) may be input to the machine-learned model, and the output (e.g., predicted) image may be compared to the HDR ground truth image. A loss function may be used to evaluate the comparison. The machine-learned model may include weights. The weights may be modified or learned during the training operation. For example, if evaluation of the loss function results in a loss greater than a threshold, the weights may be modified and training continues. If evaluation of the loss function results in a loss equal to or less than the threshold, training terminates. Training may include using supervised learning methods.

[0034] Continuing with the example machine learning encoder-decoder architecture with two encoders, training may include inputting a first training image (e.g., a low-resolution image processed image) to an encoder and a second training image (e.g., a high-resolution raw image) to another encoder. The first training image and the second training image may be encoded, and associated channels may be added together. The added channel may be decoded to predict an image (e.g., an HDR image). The predicted image may be compared to an HDR ground truth image. A loss function may be used to evaluate this comparison, and weights associated with the encoder(s) and the decoder may be modified based on the evaluation of the loss function. If the loss function evaluation results in a loss below a threshold, training of the machine learning encoder-decoder architecture with two encoders is terminated. In an exemplary implementation, a combination of multiple first training images, second training images, and HDR ground truth images may be used to train the machine learning encoder-decoder architecture with two encoders.

[0035] FIG. 3 illustrates a method of operating a wearable device to generate HDR images according to an exemplary implementation. As shown in FIG. 3, in step S305, a plurality of images having a first number of pixels are captured. All images of the plurality of images may have the same first number of pixels. For example, the plurality of images may be captured as an image burst on the wearable device. The image burst may be a plurality of RAW images (e.g., unprocessed images, uncompressed images, low-quality images, underexposed images, noisy images, and / or other such images). All images of the plurality of images may have a first resolution. The first resolution may be a relatively high resolution (e.g., 12 megapixels). The first resolution may be a variable setting associated with the camera, although this setting may not be changeable during capture of the image burst. Thus, each image of the image burst may have the same resolution (e.g., number of pixels).

[0036] In step S310, an image having a second number of pixels is generated based on the plurality of images as a first image. The first image can be generated based on all of the plurality of images. Generating the image can include some image processing of the plurality of images. Generating the image can include aligning and merging the plurality of images. Aligning and merging the plurality of images can remove noise from the plurality of images. The plurality of images can be aligned using a Gaussian pyramid technique that progresses from coarse to finer alignment. Each level of the pyramid can be downsampled. Aligning the plurality of images can prevent motion blur or ghosting.

[0037] Multiple images can be merged to generate a single image, and noise associated with the resulting image relative to the reference image can be reduced by including similar information from other aligned images. Merging can include using a variation of a Wiener filter on each tile (e.g., portion of an image) across the time dimension. Merging can improve color accuracy in dark areas of the resulting image. The multiple images can be downsampled during the alignment and merging operations. The multiple images can be compressed before, during, and / or after the alignment and merging operations. Downsampling and / or compressing the multiple images and / or images can generate an image having a second number of pixels. The second number of pixels can be smaller than the first number of pixels. Thus, the image having the second number of pixels can have a lower resolution (e.g., 3 megapixels) compared to the multiple images.

[0038] In step S315, one of the multiple images is selected as the second image. The second image may have the same number of pixels as each of the multiple images. In other words, the second image may have a relatively high resolution (e.g., 12 megapixels). The second image may be an unprocessed or RAW image. The second image 120 may be selected based on a setting of the wearable device. For example, a preconfigured setting or a user setting may be stored in memory. The setting may indicate, for example, that the first, last, middle, and / or other image in the time sequence of the multiple images should be selected. Alternatively, the setting may include a randomly selected image in the time sequence of the multiple images. Alternatively, the setting may include determining a best image (e.g., some quality threshold) and selecting the best image in the time sequence of the multiple images.

[0039] In step S320, the first image and the second image are transmitted to the companion device. For example, the wearable device and the companion device may be communicatively coupled using wired (e.g., Ethernet) and / or wireless (e.g., Bluetooth) standards. As a result, the first image and the second image can be transmitted from the wearable device to the companion device.

[0040] 4 illustrates a method for operating a companion device to generate an HDR image according to an exemplary embodiment. As shown in FIG. 4, in step S405, a first image and a second image are received from a wearable device. The first image may have a relatively low resolution (e.g., 3 megapixels), and the second image may have a relatively high resolution (e.g., 12 megapixels). The first image may be a processed image, and the second image may be an unprocessed (e.g., unprocessed) or RAW image.

[0041] In step S410, the first image is processed. Processing the first image may include color mapping and / or tone mapping (or adjustment) of the first image. Color mapping and / or tone mapping may be an optional operation. Color mapping and / or tone mapping may be configured to map pixels to different color (e.g., RGB) values ​​based on the color and / or hue associated with the pixel and / or surrounding pixels. Color mapping and / or tone mapping may be configured to produce a more natural-looking image. Color mapping and / or tone mapping may be configured to reduce the overall dynamic range or contrast ratio of the image while preserving local contrast. Processing the first image may include other image finishing operations. For example, processing the first image may include performing lens shading correction, white balancing, demosaicing, chromatic aberration correction, sharpening, and / or the like.

[0042] In step S415, the first image is merged with the second image to generate a high dynamic range (HDR) image. For example, merging the first image with the second image can include using a machine-learned model to merge the first image with the second image. The machine-learned model can be a deep learning architecture (e.g., U-Net). The machine-learned model can be trained to generate an image having a relatively high resolution (e.g., 12 megapixels) related to the second image and a post-processed (e.g., denoised) version of the first image. In other words, the machine-learned model can be trained to generate an HDR image based on the first image and the second image.

[0043] FIG. 5A illustrates a wearable device configured to generate HDR images according to an exemplary implementation. In the example of FIG. 5A, the wearable device 110 (e.g., smart glasses) may include a computing system or at least one computing device, and it should be understood that the device represents virtually any computing device configured to perform the techniques described herein. Thus, the device may be understood to include various components that may be utilized to implement the techniques described herein, or different or future versions thereof. By way of example, the system may include a processor 505, a memory 510 (e.g., non-transitory computer-readable memory), and a camera 515. The processor 505 and the memory 510 may be coupled (e.g., communicatively coupled) by a bus.

[0044] The processor 505 may be utilized to execute instructions stored in at least one memory 510. As such, the processor 505 may implement various features and functions described herein, or additional or alternative features and functions. The processor 505 and the at least one memory 510 may be utilized for various other purposes. For example, the at least one memory 510 may represent examples of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein.

[0045] The at least one memory 510 may be configured to store data and / or information related to the device. The at least one memory 510 may be a shared resource. Thus, the at least one memory 510 may be configured to store data and / or information related to other elements in a larger system (e.g., image / video processing or wired / wireless communication). The processor 505 and the at least one memory 510 may be used together to implement the techniques described herein. As such, the techniques described herein may be implemented as code segments (e.g., software) stored in the memory 510 and executed by the processor 505. Optionally, the memory 510 may include an image processing module 210. As described above, the image processing module 210 may be configured to generate a first image and a second image based on a plurality of images. Generating the first image may include performing some image processing on the plurality of images. Generating the second image may include selecting one of the plurality of images. The second image 120 may be selected based on settings of the wearable device. For example, preconfigured settings or user settings may be stored in the memory. This setting may indicate, for example, that the first, last, middle, and / or the like in a time sequence of multiple images should be selected.

[0046] FIG. 5B illustrates a companion device configured to generate an HDR image according to an exemplary implementation. In the example of FIG. 5B, the companion device (e.g., a mobile phone) may include a computing system or at least one computing device, and it should be understood that the companion device represents virtually any computing device configured to perform the techniques described herein. Thus, the device may be understood to include various components that may be utilized to implement the techniques described herein, or different or future versions thereof. By way of example, the system may include a processor 520 and memory 525 (e.g., non-transitory computer-readable memory). The processor 520 and memory 525 may be coupled (e.g., communicatively coupled) by a bus.

[0047] The processor 520 may be utilized to execute instructions stored in at least one memory 525. As such, the processor 520 may implement various features and functions described herein, or additional or alternative features and functions. The processor 520 and the at least one memory 525 may be utilized for various other purposes. For example, the at least one memory 525 may represent examples of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein.

[0048] The at least one memory 525 may be configured to store data and / or information related to the device. The at least one memory 525 may be a shared resource. Thus, the at least one memory 525 may be configured to store data and / or information related to other elements in a larger system (e.g., image / video processing or wired / wireless communication). The processor 520 and the at least one memory 525 may be used together to implement the techniques described herein. As such, the techniques described herein may be implemented as code segments (e.g., software) stored in the memory 525 and executed by the processor 520. Optionally, the memory 525 may include an image processing module 230 and a machine learning module 235. As described above, the image processing module 230 may be configured to color map and / or tone map (or adjust) an image. The image processing module 230 may be configured to perform other image finishing operations. As described above, the image merging module 235 may be configured to generate an HDR image based on a first image (e.g., a low-resolution image) and a second image (e.g., a high-resolution image).

[0049] 6 illustrates a wearable device according to an exemplary embodiment. As illustrated in FIG. 6, wearable device 600 includes lens frame 605, lens frame 610, center frame support 615, lens element 620, lens element 625, extended side arm 630, extended side arm 635, image capture device 640 (e.g., camera), on-board computing system 645, speaker 650, and microphone 655.

[0050] Each of the frame elements 605, 610, and 615 and the extended side arms 630, 635 may be formed from a solid structure of plastic and / or metal, or from a hollow structure of a similar material to allow wiring and component interconnections to be routed internally through the wearable device 600. Other materials may be possible. At least one of the lens elements 620, 625 may be formed from any material capable of adequately displaying projected images or graphics. Each of the lens elements 620, 625 may also be sufficiently transparent to allow a user to see through the lens element. Combining these two features of the lens elements may facilitate augmented reality or head-up displays, where projected images or graphics are superimposed on the real-world view perceived by the user through the lens element.

[0051] The center frame support 615 and the extended side arms 630, 635 are configured to secure the wearable device 600 to the user's face via the user's nose and ears, respectively. Each of the extended side arms 630, 635 may be a protrusion extending away from the lens frames 605, 610, respectively, and may be positioned behind the user's ear to secure the wearable device 600 to the user. The extended side arms 630, 635 may further secure the wearable device 600 to the user by extending around the back of the user's head. Additionally or alternatively, for example, the wearable device 600 may be connected to or mounted within a head-mounted helmet structure. Other configurations of the wearable computing device are possible.

[0052] The on-board computing system 645 is shown located on the extended side arm 630 of the wearable device 600. However, the on-board computing system 645 may be provided in another portion of the wearable device 600 or may be located remotely from the wearable device 600 (e.g., the on-board computing system 645 may be connected to the wearable device 600 via a wired or wireless connection). The on-board computing system 645 may include, for example, a processor and memory. The on-board computing system 645 may be configured to receive and analyze data from the image capture device 640 (and possibly from other sensor devices) and generate images for output by the lens elements 620, 625.

[0053] Image capture device 640 may be, for example, a camera configured to capture still images and / or capture video. In the illustrated configuration, image capture device 640 is located on extended side arm 630 of wearable device 600. However, image capture device 640 may be located in other portions of wearable device 600. Image capture device 640 may be configured to capture images at various resolutions or different frame rates. Many image capture devices with small form factors, such as cameras used in cell phones or webcams, may be incorporated into examples of wearable device 600.

[0054] One image capture device 640 is shown. However, more image capture devices may be used, each configured to capture the same view or a different view. For example, image capture device 640 may face forward to capture at least a portion of the real-world view perceived by the user. This forward-facing image captured by image capture device 640 may then be used to generate an augmented reality, where computer-generated images appear to interact with or be superimposed on the real-world view perceived by the user.

[0055] Exemplary embodiments may include a non-transitory computer-readable storage medium including instructions that, when stored on the non-transitory computer-readable storage medium and executed by at least one processor, are configured to cause a computing system to perform any of the methods described above. Exemplary embodiments may include an apparatus including means for performing any of the methods described above. Exemplary embodiments may include an apparatus including at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, using the at least one processor, cause the apparatus to at least perform any of the methods described above.

[0056] A split-compute architecture may be an architecture that moves the runtime environment of an app to a remote compute endpoint or companion device, such as a phone, server, cloud, or desktop computer, often referred to as a mobile device or cell phone for simplicity. In some implementations, data sources such as IMUs and camera sensors may be streamed from the wearable device to the companion device. In some implementations, display content may be streamed from the companion device back to the wearable device. In some implementations, data streams, video / audio streams, bidirectional control streams, and / or the like may be communicated between the wearable device and the companion device. In some implementations, a split-compute architecture may enable leveraging low-power MCU-based systems, since some of the computing and rendering may not occur on the wearable device itself. In some implementations, this may enable power and ID savings, meeting design constraints, etc. Codecs and networking may allow the required networking bandwidth to be sustained in a low-power manner. In some implementations, a wearable device may be connected to multiple companion devices at once. In some implementations, different companion devices may provide different services. In some implementations, as low-latency, high-bandwidth 5G connections become mainstream, the companion device may reside in the cloud. In some implementations, the wearable device may communicate with the companion device via a well-defined protocol. This architecture may be platform-independent.

[0057] 7 illustrates examples of a computing device 700 and a mobile computing device 750 that may be used with the techniques described herein (e.g., to implement wearable device 110 and companion device 125). Computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 that connects to memory 704 and a high-speed expansion port 710, and a low-speed interface 712 that connects to a low-speed bus 714 and storage device 706. Components 702, 704, 706, 708, 710, and 712 are interconnected using various buses and may be mounted on a common motherboard or in other manners as needed. Processor 702 can process instructions for execution within computing device 700, including instructions stored in memory 704 or storage device 706, to display graphical information for a GUI on an external input / output device, such as a display 716, connected to high-speed interface 708. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as needed. Also, multiple computing devices 700 may be connected together (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system), with each device providing a portion of the required operations.

[0058] The memory 704 stores information within the computing device 700. In one implementation, the memory 704 is a volatile memory unit(s). In other implementations, the memory 704 is a non-volatile memory unit(s). The memory 704 may also be other forms of computer-readable media, such as a magnetic disk or optical disk.

[0059] The storage device 706 can provide mass storage for the computing device 700. In one embodiment, the storage device 706 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or any number of devices, including a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. A computer program product can be tangibly embodied on an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, examples of which include memory 704, the storage device 706, or memory on the processor 702.

[0060] The high-speed controller 708 manages the bandwidth-intensive operations of the computing device 700, while the low-speed controller 712 manages the less bandwidth-intensive operations. Such an allocation of functions is merely an example. In one embodiment, the high-speed controller 708 is coupled to the memory 704, the display 716 (e.g., via a graphics processor or accelerator), and to a high-speed expansion port 710 that may accept various expansion cards (not shown). In an embodiment, the low-speed controller 712 is coupled to the storage device 706 and the low-speed expansion port 714. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet). The low-speed expansion port may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or may be coupled to a network device, such as a switch or router, for example, via a network adapter.

[0061] Computing device 700 may be implemented in a number of different forms, as shown. For example, it may be implemented as a standard server 720, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 724. Additionally, it may be implemented in a personal computer, such as a laptop computer 722. Alternatively, the components of computing device 700 may be combined with other components in a mobile device (not shown), such as device 750. Each such device may include one or more of computing devices 700, 750, and the entire system may consist of multiple computing devices 700, 750 in communication with each other.

[0062] Computing device 750 includes, among other components, a processor 752, memory 764, input / output devices such as a display 754, a communications interface 766, and a transceiver 768. Device 750 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 750, 752, 764, 754, 766, and 768 are interconnected using various buses, and some of the components may be mounted on a common motherboard or in other manners as desired.

[0063] The processor 752 can execute instructions within the computing device 750, including instructions stored in the memory 764. The processor can be implemented as a chipset of chips including separate analog and digital processors. The processor may provide, for example, coordination of other components of the device 750, including control of a user interface, applications run by the device 750, and wireless communication by the device 750.

[0064] Processor 752 may communicate with a user via a control interface 758 and a display interface 756 coupled to a display 754. Display 754 may be, for example, a TFT LCD (thin film transistor liquid crystal display), an LED (light emitting diode), or an OLED (organic light emitting diode) display, or other suitable display technology. Display interface 756 may include appropriate circuitry for driving display 754 to present graphical and other information to a user. Control interface 758 may receive commands from a user and convert the commands for submission to processor 752. Additionally, an external interface 762 may be provided in communication with processor 752 to enable short-range communication between device 750 and other devices. External interface 762 may, for example, provide for wired communication in some implementations or wireless communication in other implementations, and multiple interfaces may be used.

[0065] Memory 764 stores information within computing device 750. Memory 764 may be implemented as one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. Expansion memory 774 may also be provided and may be connected to device 750 via expansion interface 772. Expansion interface 772 may include, for example, a SIMM (Single In-Line Memory Module) card interface. Such expansion memory 774 may provide additional storage space for device 750 or may store applications or other information for device 750. Specifically, expansion memory 774 may include instructions for performing or supplementing the above-described processes and may also include secure information. Thus, for example, expansion memory 774 may be provided as a security module for device 750 and may be programmed with instructions that enable secure use of device 750. Furthermore, secure applications may be provided via SIMM cards along with additional information, such as carrying identifying information on the SIMM card in an unhackable manner.

[0066] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, examples of which include memory 764, expansion memory 774, or memory on processor 752. The information carrier may be received, for example, via transceiver 768 or external interface 762.

[0067] Device 750 may communicate wirelessly via a communication interface 766, which may include digital signal processing circuitry if necessary. Communication interface 766 may provide for communication in various modes or protocols, examples of which include GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, via a radio frequency transceiver 768. Additionally, short-range communication may occur, such as by using Bluetooth, WiFi, or other such transceivers (not shown). Additionally, a GPS (Global Positioning System) receiver module 770 may provide additional navigation- and location-related wireless data to device 750, which may be used as needed by applications executing on device 750.

[0068] Device 750 may also perform voice communications using an audio codec 760, which may receive voice information from a user and convert it into usable digital information. Audio codec 760 may also generate sounds that are heard by the user, such as through a speaker (e.g., in the handset of device 750). Such sounds may include sounds from voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications running on device 750.

[0069] Computing device 750 may be implemented in a number of different forms, as shown in the figure. For example, computing device 750 may be implemented as a mobile phone 780. Computing device 750 may also be implemented as part of a smartphone 782, personal digital assistant, or other similar mobile device.

[0070] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include execution in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be specialized or general-purpose, coupled to receive data and instructions from the storage system, and to transmit data and instructions to the storage system.

[0071] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0072] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer that has a display device (such as an LED (light-emitting diode), OLED (organic LED), or LCD (liquid crystal display) monitor / screen) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0073] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., a data server, etc.), or that includes middleware components (e.g., an application server), or that includes front-end components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0074] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0075] In some implementations, the illustrated computing device may include sensors that interface with the AR headset / HMD device 790 to generate an augmented environment for viewing content inserted within a physical space. For example, one or more sensors included on the computing device 750 or other computing devices illustrated may provide input to the AR headset 790 or, generally, to the AR space. The sensors may include, but are not limited to, a touchscreen, an accelerometer, a gyroscope, a pressure sensor, a biometric sensor, a temperature sensor, a humidity sensor, and an ambient light sensor. The computing device 750 may use the sensors to determine the absolute position and / or detected rotation of the computing device within the AR space, which may then be used as input to the AR space. For example, the computing device 750 may be embedded within the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, or the like. When embedded in the AR space, a user's positioning of the computing device / virtual object may allow the user to position the computing device to view the virtual object in a particular way within the AR space. For example, if the virtual object represents a laser pointer, the user may manipulate the computing device as if it were an actual laser pointer. A user can move the computing device left and right, up and down, in a circle, etc. to use the device in a manner similar to using a laser pointer. In some implementations, a user can use a virtual laser pointer to aim at a target location.

[0076] In some implementations, one or more input devices included on or connected to computing device 750 can be used as input to the AR space. The input devices can include, but are not limited to, a touchscreen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, earphones or earphones with input capabilities, a game controller, or other connectable input devices. A user interacting with an input device included on computing device 750 when the computing device is integrated into the AR space can cause certain actions to occur in the AR space.

[0077] In some implementations, the touchscreen of the computing device 750 can be rendered as a touchpad in the AR space. A user can interact with the touchscreen of the computing device 750. The interaction is rendered in the AR headset 790, for example, as movements on a touchpad that are rendered in the AR space. The rendered movements can control virtual objects in the AR space.

[0078] In some implementations, one or more output devices included on the computing device 750 can provide output and / or feedback to a user of the AR headset 790 within the AR space. The output and feedback may be visual, tactical, or audio. The output and / or feedback may include, but is not limited to, vibration, turning one or more lights or strobes on and off or flashing and / or flashing, sounding an alarm, chiming, playing a song, and playing an audio file. Output devices may include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light-emitting diodes (LEDs), strobes, and speakers.

[0079] In some implementations, the computing device 750 may be displayed as another object in the computer-generated 3D environment. A user's interactions with the computing device 750 (e.g., rotating, shaking, touching, or swiping a finger on the touchscreen) may be interpreted as interactions with objects in the AR space. In the example of a laser pointer in an AR space, the computing device 750 is displayed as a virtual laser pointer in the computer-generated 3D environment. As the user manipulates the computing device 750, the user in the AR space sees the movement of the laser pointer. The user receives feedback from their interactions with the computing device 750 in the AR environment on the computing device 750 or on the AR headset 790. The user's interactions with the computing device may be translated into interactions with a user interface generated in the AR environment for the controllable device.

[0080] In some implementations, computing device 750 may include a touchscreen. For example, a user may interact with the touchscreen to interact with a user interface for the controllable device. For example, the touchscreen may include user interface elements, such as sliders, that can control characteristics of the controllable device.

[0081] Computing device 700 is intended to represent various types of digital computers and devices, including, but not limited to, laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 750 is intended to represent various types of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the implementation of the invention(s) described and / or claimed herein.

[0082] Exemplary embodiments may be embodied as a non-transitory computer-readable storage medium including instructions, the instructions being stored on the non-transitory computer-readable storage medium and configured, when executed by at least one processor, to cause a computing system to perform certain operations and / or steps in accordance with the above-described embodiments. Exemplary embodiments may be embodied as an apparatus including means for performing certain operations and / or steps in accordance with the above-described embodiments. Exemplary embodiments may be embodied as an apparatus including at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured, using the at least one processor, to cause the apparatus to perform at least certain operations and / or steps in accordance with the above-described embodiments.

[0083] Although a number of embodiments have been described, it will nevertheless be understood that various modifications may be made without departing from the spirit and scope of the present disclosure.

[0084] Additionally, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve desired results. Additionally, other steps may be provided in or eliminated from the described flows, and other components may be added to or removed from the described systems. Accordingly, other embodiments are within the scope of the following claims.

[0085] In addition to the above, the system, program, or functionality described herein may provide users with controls that allow them to choose both whether and when to enable the collection of user information (e.g., information regarding the user's social networks, social actions, or activities, occupation, user preferences, or the user's current location) and whether to send content or communications from the server to the user. Additionally, certain data may be processed in one or more ways so that personally identifiable information is removed before it is stored or used. For example, the user's identifying information may be processed so that personally identifiable information about the user cannot be determined, or if location information is obtained (e.g., to the city, zip code, or state level), the user's geographic location may be generalized so that the user's specific location cannot be determined. Thus, users may control what information is collected about them, how that information is used, and what information is provided to them.

[0086] As described herein, while certain features of the described embodiments have been illustrated, numerous modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and variations that fall within the scope of the embodiments. They are presented by way of example only, not limitation, and it should be understood that various changes in form and detail may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination except mutually exclusive combinations. The embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different embodiments described.

[0087] The exemplary embodiments may include various modifications and alternative forms, which are shown by way of example in the drawings and described in detail herein. It is to be understood, however, that there is no intention to limit the exemplary embodiments to the particular forms disclosed, but rather that the exemplary embodiments cover all modifications, equivalents, and alternatives falling within the scope of the claims. Like numerals refer to like components throughout the description of the figures.

[0088] Some of the above exemplary embodiments are described as a process or method that is depicted as a flowchart. While the flowcharts describe operations as sequential, many of the operations may occur in parallel, concurrently, or simultaneously. The order of operations may also be rearranged. A process may terminate upon completion of its operations, or may have additional steps not included in the figures. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0089] The above methods, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine- or computer-readable medium such as a storage medium. A processor(s) may perform the necessary tasks.

[0090] Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments, however, example embodiments may be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.

[0091] Terms such as first, second, etc. may be used herein to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0092] When an element is referred to as being connected or coupled to other elements, it will be understood that the element may be directly connected or coupled to the other elements, or that intervening elements may be present. In contrast, when an element is referred to as being directly connected to or directly coupled to other elements, there are no intervening elements present. Other words used to indicate the relationship between elements should be interpreted similarly (e.g., between and directly between, adjacent and directly adjacent, etc.).

[0093] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms a, an, and the are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is to be further understood that the terms "comprises," "comprising," "includes," and / or "including," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0094] It should also be noted that in some alternative implementations, the functions / acts shown may occur out of the order noted in the figures. For example, two figures shown in succession may, in fact, be executed concurrently or may sometimes be executed in the reverse order, depending upon the functions / acts involved.

[0095] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary embodiments belong. Furthermore, it will be understood that terms (e.g., as defined in commonly used dictionaries) should be interpreted to have a meaning consistent with their meaning in the context of the relevant art, and not in an idealized or overly formal sense unless expressly defined herein.

[0096] Portions of the above exemplary embodiments and corresponding detailed description are presented in terms of software, or algorithms, and symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the ways in which those skilled in the art effectively convey the substance of their work to others skilled in the art. An algorithm, as the term is used herein, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. These steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0097] In the above exemplary embodiments, references to symbolic representations (e.g., in the form of flowcharts) of acts and operations that may be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types, and may be described and / or implemented using existing hardware in existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate array (FPGA) computers, etc.

[0098] It should be recognized, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specified, or as will be apparent from the description, terms such as processing, operating, calculating, determining, displaying, etc., refer to the actions and processes of a computer system or similar electronic computing device that manipulate and convert data represented as physical, electronic quantities in the computer system's registers and memory into other data similarly represented as physical quantities in the computer system's memory or registers, or other such information storage, transmission, or display device.

[0099] It should also be noted that software implementations of the exemplary embodiments are typically encoded on some form of non-transitory program storage medium or implemented via some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or hard drive) or optical (e.g., a compact disk read-only memory, or CD ROM), and may be read-only or random-access. Similarly, the transmission medium may be twisted wire pair, coaxial cable, optical fiber, or other suitable transmission medium known in the art. The exemplary embodiments are not limited by these aspects of any given implementation.

[0100] Finally, it should also be noted that while the appended claims set forth particular combinations of features described herein, the scope of the disclosure is broadened to encompass any combination of features or embodiments disclosed herein, and is not limited to the particular combinations claimed below, regardless of whether that particular combination is specifically recited in the appended claims at this time.

Claims

1. A non-transitory computer-readable storage medium including instructions that, when stored on the non-transitory computer-readable storage medium and executed by at least one processor, cause a wearable device to: capturing, by the wearable device, a plurality of images having a first number of pixels; processing, by the wearable device, the plurality of images to generate a first image having a second number of pixels that is less than the first number of pixels; selecting, by the wearable device, one of the images having the first number of pixels as a second image based on a setting of the wearable device; transmitting the first image and the second image to a companion device so as to generate a high dynamic range (HDR) image by the wearable device; 1. A non-transitory computer-readable storage medium configured to cause execution of

2. The instructions may further include causing the wearable device to: controlling, by the wearable device, the companion device to generate the HDR image based on the first image and the second image; The non-transitory computer-readable storage medium of claim 1 ,

3. 3. The non-transitory computer-readable storage medium of claim 1, wherein the capturing of the plurality of images is an image burst that includes capturing a series of images in response to a single trigger.

4. 4. The non-transitory computer-readable storage medium of claim 1, wherein the processing of the plurality of images to generate the first image includes compressing at least one of the first image and each of the plurality of images.

5. 4. The non-transitory computer-readable storage medium of claim 1, wherein the processing of the plurality of images to generate the first image includes downsampling at least one of the first image and each of the plurality of images.

6. The instructions may further include causing the wearable device to:

6. The non-transitory computer-readable storage medium of claim 1, wherein a signal is received from the companion device, and wherein the capturing of the plurality of images is in response to receiving the signal.

7. The processing of the plurality of images to generate the first image includes: reducing the number of pixels in each of the plurality of images to the second number of pixels; aligning corresponding pixels of the plurality of images; merging the plurality of images to generate the first image; 7. The non-transitory computer-readable storage medium of claim 1, comprising:

8. 8. The non-transitory computer-readable storage medium of claim 1, wherein the processing of the plurality of images to generate the first image comprises tone mapping and color adjusting the image having the second number of pixels.

9. A non-transitory computer-readable storage medium including instructions that, when stored on the non-transitory computer-readable storage medium and executed by at least one processor, cause a companion device to: The instructions are configured to cause the companion device to receive, from the wearable device, a first image having a first resolution and a second image having a second resolution, the first resolution being smaller than the second resolution, and the first image and the second image being generated based on an image burst including capturing a series of images in response to a trigger, and the instructions further cause the companion device to: processing, by the companion device, the first image; merging, by the companion device, the first image with the second image to generate a high dynamic range (HDR) image; 1. A non-transitory computer-readable storage medium configured to cause execution of

10. The merging of the first image with the second image comprises: inputting the first image and the second image into a machine-learned model; generating the HDR image based on the first image together with the second image using the machine-learned model; 10. The non-transitory computer-readable storage medium of claim 9, comprising:

11. The non-transitory computer-readable storage medium of claim 10 , wherein the machine-learned model is trained using HDR ground truth images.

12. The instructions may further include causing the companion device to: controlling, by the companion device, the wearable device to initiate the generation of the HDR image based on the first image and the second image; The non-transitory computer-readable storage medium according to any one of claims 9 to 11, which causes the execution of the above.

13. The instructions may further include causing the companion device to:

13. The non-transitory computer-readable storage medium of claim 9, further comprising: transmitting a signal to the wearable device, the signal causing the wearable device to trigger the image burst.

14. 1. A system including a wearable device and a companion device, at least one processor; at least one memory containing computer program code; Equipped with The at least one memory and the computer program code, when used with the at least one processor, cause the system to: capturing, by the wearable device, a plurality of images, each having a first resolution; and processing the plurality of images by the wearable device to generate a first image having a second resolution, the second resolution being smaller than the first resolution, and the at least one memory and the computer program code are further configured to cause the system to: selecting, by the wearable device, a second image from the plurality of images having the first resolution based on a setting of the wearable device; transmitting, by the wearable device, the first image and the second image to the companion device; processing, by the companion device, the first image; merging, by the companion device, the processed first image with the second image to generate a high dynamic range (HDR) image; A system configured to cause

15. The processing of the plurality of images to generate the first image includes: reducing the resolution of each of the plurality of images to the second resolution; aligning corresponding pixels of the plurality of images; merging the plurality of images to generate the first image; The system of claim 14 , comprising:

16. 16. The system of claim 14 or 15, wherein said processing said first image comprises tone mapping and color adjusting said first image.

17. The merging of the first image with the second image comprises: inputting the processed first image and the second image into a machine-learned model; generating the HDR image based on the processed first image together with the second image using the machine-learned model; The system according to any one of claims 14 to 16, comprising:

18. The system of claim 17 , wherein the machine-learned model is trained using HDR ground truth images.

19. controlling, by the companion device, the wearable device to initiate the generation of the HDR image based on the first image and the second image; The system of any one of claims 14 to 18, further comprising:

20. communicating a signal from the companion device to the wearable device to trigger the wearable device to capture the plurality of images; The system of any one of claims 14 to 19, further comprising:

21. controlling, by the wearable device, the companion device to generate the HDR image based on the first image and the second image; The system of any one of claims 14 to 20, further comprising:

22. The system of any one of claims 14 to 21, wherein said capturing said plurality of images is an image burst comprising capturing a series of images in response to a single trigger.

23. Capturing, by a wearable device, a plurality of images having a first number of pixels; processing, by the wearable device, the plurality of images to generate a first image having a second number of pixels that is less than the first number of pixels; selecting, by the wearable device, one of the images having the first number of pixels as a second image based on a setting of the wearable device; transmitting the first image and the second image to a companion device so as to generate a high dynamic range (HDR) image by the wearable device; A method comprising:

24. A wearable device, at least one processor; at least one memory containing computer program code; Equipped with The at least one memory and the computer program code are configured to cause the wearable device, using the at least one processor, to: capturing, by the wearable device, a plurality of images having a first number of pixels; processing, by the wearable device, the plurality of images to generate a first image having a second number of pixels that is less than the first number of pixels; selecting, by the wearable device, one of the images having the first number of pixels as a second image based on a setting of the wearable device; transmitting the first image and the second image to a companion device so as to generate a high dynamic range (HDR) image by the wearable device; A wearable device configured to:

25. A wearable device, capturing, by the wearable device, a plurality of images having a first number of pixels; processing, by the wearable device, the plurality of images to generate a first image having a second number of pixels that is less than the first number of pixels; selecting, by the wearable device, one of the images having the first number of pixels as a second image based on a setting of the wearable device; transmitting the first image and the second image to a companion device so as to generate a high dynamic range (HDR) image by the wearable device; 10. A wearable device comprising: a means for:

26. 1. A method comprising: receiving, by a companion device, from a wearable device, a first image having a first resolution and a second image having a second resolution, the first resolution being smaller than the second resolution, the first image and the second image being generated based on an image burst comprising capturing a series of images in response to a trigger, the method further comprising: processing, by the companion device, the first image; merging, by the companion device, the first image with the second image to generate a high dynamic range (HDR) image; A method comprising:

27. a companion device, at least one processor; at least one memory containing computer program code; Equipped with The at least one memory and the computer program code, using the at least one processor, cause the companion device to: and a second image having a second resolution, the first resolution being smaller than the second resolution, the first image and the second image being generated based on an image burst including capturing a series of images in response to a trigger; and the at least one memory and the computer program code are further configured to cause the companion device to: processing, by the companion device, the first image; merging, by the companion device, the first image with the second image to generate a high dynamic range (HDR) image; a companion device configured to cause the

28. 1. A companion device, comprising: means for receiving, by the companion device, from the wearable device, a first image having a first resolution and a second image having a second resolution, the first resolution being smaller than the second resolution, the first image and the second image being generated based on an image burst comprising capturing a series of images in response to a trigger, the companion device further comprising: processing, by the companion device, the first image; merging, by the companion device, the first image with the second image to generate a high dynamic range (HDR) image; a companion device comprising means for:

Images