High dynamic range (HDR) photography with split computation on wearable devices

Abstract

A method includes capturing, by a wearable device, a plurality of images each having a first resolution; processing, by the wearable device, the plurality of images to generate a first image having a second resolution, the second resolution being less than the first resolution; selecting, by the wearable device, a second image from the plurality of images having the first resolution based on the settings of the wearable device; and transmitting, by the wearable device, the first image and the second image to a companion device. Further, the first image is processed by the companion device and the processed first image is merged with the second image by the companion device to generate a high dynamic range (HDR) image.

Description

Technical Field

[0001] Embodiments relate to a digital photography process for images captured using a wearable device. Background Art

[0002] High dynamic range (HDR) photography is standard for flagship mobile phones. The goal involves designing a pipeline that takes a burst of images (e.g., about ten frames, sometimes underexposed) as input. The burst of images is then aligned, merged, and denoised to generate one HDR image. The HDR image may be tone mapped and / or other image post-processing may be performed. HDR may then be made accessible to users of mobile devices. Summary of the invention

[0003] The wearable device may be communicatively coupled to the companion device to split the computing operation. In an example implementation, the wearable device may capture an image burst. The image burst may be used to generate a first image having a smaller resolution than an image in the image burst. The wearable device may select a second image from the image burst. The first image and the second image may be transmitted to the companion device. The companion device may generate an HDR image by merging the first image and the second image.

[0004] In general terms, a device, a system, a non-transitory computer-readable medium (having computer executable program code stored thereon, which can be executed on a computer system) and / or a method can utilize a method including the following operations to perform a process: capturing multiple images having a first number of pixels by a wearable device; processing the multiple images by the wearable device to generate a first image having a second number of pixels less than the first number of pixels; selecting one of the multiple images having the first number of pixels as a second image based on a setting of the wearable device; and transmitting the first image and the second image by the wearable device to a companion device to generate a high dynamic range (HDR) image.

[0005] Implementations may include one or more of the following features or any combination thereof. For example, the method may further include controlling, by the wearable device, a companion device to generate an HDR image based on the first image and the second image. The capture of multiple images may be an image burst including capturing a series of images in response to a trigger. Processing multiple images to generate a first image may include compressing at least one of the first image and each of the multiple images. Processing multiple images to generate a first image may include downsampling at least one of the first image and each of the multiple images.

[0006] The method may further include receiving a signal from a companion device, wherein the capturing of the plurality of images is in response to receiving the signal. Processing the plurality of images to generate a first image may include reducing the number of pixels in each of the plurality of images to a second number of pixels, aligning corresponding pixels of the plurality of images, and merging the plurality of images to generate the first image. Processing the plurality of images to generate the first image may include tone mapping and color tuning the images having the second number of pixels.

[0007] In another general aspect, a device, a system, a non-transitory computer-readable medium (having computer executable program code stored thereon, which can be executed on a computer system) and / or a method can perform a process using a method including the following operations: receiving, by a companion device, a first image having a first resolution and a second image having a second resolution from a wearable device, the first resolution being smaller than the second resolution, the first image and the second image being generated based on an image burst including capturing a series of images in response to a trigger; processing, by the companion device, the first image; and merging, by the companion device, the first image with the second image to generate a high dynamic range (HDR) image.

[0008] Implementations may include one or more of the following features or any combination thereof. For example, the merging of the first image and the second image may include inputting the first image and the second image into a machine learning model, and using the machine learning model to generate an HDR image based on the first image and the second image. The machine learning model may be trained using an HDR true value image. The method may further include controlling the wearable device by a matching device to initiate the generation of an HDR image based on the first image and the second image. The method may further include transmitting a signal to the wearable device, wherein the signal causes the wearable device to trigger a burst of images.

[0009] In another general aspect, a device, a system, a non-transitory computer-readable medium (having computer executable program code stored thereon, which can be executed on a computer system) and / or a method can utilize a method including the following operations to perform a process: capturing a plurality of images each having a first resolution by a wearable device; processing the plurality of images by the wearable device to generate a first image having a second resolution, the second resolution being smaller than the first resolution; selecting a second image from the plurality of images having the first resolution by the wearable device based on a setting of the wearable device; transmitting the first image and the second image to a companion device by the wearable device; processing the first image by the companion device; and merging the processed first image with the second image by the companion device to generate a high dynamic range (HDR) image.

[0010] Implementations may include one or more of the following features or any combination thereof. For example, processing a plurality of images to generate a first image may include reducing the resolution of each of the plurality of images to a second resolution, aligning corresponding pixels of the plurality of images, and merging the plurality of images to generate the first image. Processing the first image may include tone mapping and color tuning the first image. Merging the first image with the second image may include inputting the processed first image and the second image into a machine learning model, and generating an HDR image based on the processed first image and the second image using the machine learning model. The machine learning model may be trained using an HDR true value image.

[0011] The method may further include controlling the wearable device by the accessory device to initiate the generation of an HDR image based on the first image and the second image. The method may further include transmitting a signal from the accessory device to the wearable device, the signal causing the wearable device to trigger the capture of multiple images. The method may further include controlling the accessory device by the wearable device to generate an HDR image based on the first image and the second image. The capture of multiple images may be an image burst including capturing a series of images in response to a trigger. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Example embodiments will be more fully understood from the detailed description given herein below and the accompanying drawings, in which like elements are represented by like reference numerals, which are given by way of illustration only and thus do not limit example embodiments, and in which:

[0013] Figure 1 A system for generating an HDR image according to an example implementation is shown.

[0014] Figure 2 A system for generating an HDR image according to an example implementation is shown.

[0015] Figure 3 A method of operating a wearable device to generate an HDR image according to an example implementation is shown.

[0016] Figure 4 A method of operating a supporting apparatus to generate an HDR image according to an example implementation is shown.

[0017] Figure 5A A wearable device configured to generate an HDR image according to an example implementation is shown.

[0018] Figure 5B A supporting apparatus configured to generate an HDR image according to an example implementation is shown.

[0019] Figure 6A wearable device according to an example implementation is shown.

[0020] Figure 7 Examples of a computer device and a mobile computer device according to at least one example embodiment are shown.

[0021] It should be noted that these drawings are intended to illustrate the general characteristics of the methods and / or structures utilized in certain example embodiments and to supplement the written description provided below. However, these drawings are not drawn to scale and do not accurately reflect the precise structure or performance characteristics of any given embodiment and should not be interpreted as defining or limiting the range of values ​​or properties covered by the example embodiments. For example, the positioning of modules and / or structural elements may be reduced or enlarged for clarity. The use of similar or identical reference numerals in the various drawings is intended to indicate the presence of similar or identical elements or features. DETAILED DESCRIPTION

[0022] Generating HDR images on a wearable device (e.g., smart glasses) that includes a camera may be difficult. For example, the wearable device may have a limited computational budget. In other words, executing the HDR processing pipeline may take too much time when using a wearable device. For example, the processing time on a wearable device may be ten times the standard processing time on a mobile phone. Therefore, from the user's perspective, using the same processing pipeline on a wearable device as is typically used on a mobile phone may be undesirable.

[0023] Further, transmitting the burst of images from the wearable device to a mobile phone capable of generating HDR images may take too much time per burst of images (e.g., greater than 30 seconds). In addition, transmitting the burst of images from the wearable device to the mobile phone may otherwise use excessive resources, thereby causing a reduction in the charge on the battery of the wearable device. Reducing the charge on the battery of the wearable device may render the wearable device unusable after capturing some HDR images. Wearable device power constraints (e.g., design criteria) may include ensuring that the battery charge is sufficient to enable the capture and processing of images on the wearable device to continue for a predefined number of image captures (e.g., greater than 200).

[0024] As discussed above, there may be problems using current HDR image processing techniques on wearable devices. Example implementations use a split computation process on a wearable device to address these issues. The split computation process on the wearable device may include performing a portion of the image processing on the wearable device and performing a portion of the image processing on a companion device (e.g., a companion device communicatively coupled to the wearable device). For example, in an example implementation, alignment and denoising may be performed by the wearable device on an image having a first (e.g., lower) resolution based on an image having a second (e.g., higher) resolution (e.g., number of pixels). Additional processing (e.g., tone mapping) may be performed on the image having the first (e.g., lower) resolution on the companion device. The image having the second (e.g., higher) resolution may then be merged with the image having the first (e.g., lower) resolution (after processing) to generate an HDR image.

[0025] Figure 1 A system for generating an HDR image according to an example implementation is shown. Figure 1 As shown, the system includes a user 105, a wearable device 110, and a supporting device 125. Figure 1 Also shown in the figure is a first image 115, a second image 120, and an HDR image 130. The wearable device 110 may be configured to generate the first image 115 and the second image 120. The wearable device 110 may be configured to transmit the first image 115 and the second image 120 to the accessory device 125. The accessory device 125 may be configured to generate the HDR image 130 based on the first image 115 and the second image 120.

[0026] The wearable device 110 may be, for example, a smart glasses device (e.g., an AR glasses device), a head mounted display (HMD), an AR / VR device, a wearable computing device, etc. The user 105 is viewing a real-world view from any direction. The wearable device 110 may be configured to generate a burst of images of the real world (e.g., about ten frames, sometimes underexposed). The first image 115 and the second image 120 may be generated based on a burst of images. For example, the wearable device 110 may capture a burst of images as a plurality of images having a relatively high resolution (e.g., 12 megapixels). However, as discussed above, it may not be desirable to generate an HDR image from a plurality of images having a relatively high resolution on the wearable device 110. Therefore, the wearable device 110 may be configured to compress a plurality of images having a relatively high resolution (e.g., reduce the number of pixels) to a lower resolution (e.g., 3 megapixels). The plurality of images having a lower resolution may then be aligned, merged, and / or denoised (sometimes merging and denoising images may be a process) to generate the first image 115. The resolution (e.g., number of pixels) of the low-resolution second image can be large enough to ensure that a visually (e.g., visually to a user) high-quality HDR image can be generated, but small enough to minimize resource (e.g., processing, memory, battery) usage. The second image 120 can be selected from a plurality of images having a relatively high resolution (the second image 120 can sometimes be referred to as a noisy image because the image has not been processed to remove noise associated with the image sensor used to capture the image). The second image 120 can be selected based on a setting of the wearable device. For example, a preconfigured setting or a user setting can be stored in a memory. The setting can indicate, for example, that the first image, the last image, an intermediate image, and / or the like in a time sequence of a plurality of images (e.g., a burst of images) having a relatively high resolution is to be selected.

[0027] The supporting device 125 can be configured to receive (e.g., via a wired and / or wireless connection) a first image 115 and a second image 120. The first image 115 can be further processed by the supporting device 125. For example, the first image can be tone mapped. The supporting device 125 can be configured to generate an HDR image 130 based on the first image 115 (e.g., the further processed first image 115) and the second image 120. For example, a machine learning model can be used to merge the first image 115 and the second image 120. The machine learning model can be a deep learning architecture (e.g., a trained neural network, U-Net, etc.). The machine learning model can be trained to generate an image (e.g., an HDR image) having a relatively high resolution (e.g., 12 megapixels) of the second image 120 and post-processing (e.g., denoising) of the first image 115. In other words, the machine learning model can be trained to generate an HDR image based on the first image 115 and the second image 120.

[0028] Figure 2 A system for generating an HDR image according to an example implementation is shown. Figure 2 As shown, the system includes a wearable device 110 and a matching device 125. The wearable device includes multiple images 205, an image processing module 210, a first image 215-1 and a second image 220-1. The matching device includes a first image 215-2, a second image 220-2, an image processing module 230, an image merging module 235 and an HDR image 240.

[0029] The wearable device 110 may include a camera (e.g., an image sensor) configured to capture high resolution (e.g., 12 or more megapixels). The camera may be configured to capture images as an image burst. The image burst may include capturing a series of images in response to a trigger (e.g., a button press, a shutter press, an instruction). The image burst may be a plurality of raw (e.g., unprocessed, uncompressed, low quality, underexposed, noisy, and / or other) images. The number of images in the image burst may be variable and based on the amount of light in the environment, the dynamic range of the scene, and / or other. The image burst may be stored in a memory (e.g., a cache) as a plurality of images 205. The wearable device 110 may be configured to control the companion device 125 (e.g., transmit, signal, instruction, and / or data) to generate an HDR image.

[0030] The image processing module 210 may be configured to generate a first image 215-1 and a second image 220-1 based on the plurality of images 205. Generating the first image 215-1 may include performing some image processing on the plurality of images 205. Generating the second image 220-1 may include selecting one of the plurality of images 205. The second image 120 may be selected based on a setting of the wearable device. For example, a preconfigured setting or a user setting may be stored in a memory. The setting may indicate, for example, that the first image, the last image, an intermediate image, and / or the like in the time sequence of the plurality of images 205 is to be selected.

[0031] The image processing module 210 can be configured to compress the multiple images 205 (e.g., reduce the number of pixels). The image processing module 210 can use an image resizing algorithm to compress the multiple images 205. The image resizing algorithm can proportionally reduce the number of pixels per row (width) and the number of pixels per column (height). For example, the width of each image can be reduced to n pixels, and an image resizing programming function such as resize (e.g., resize(n, 0)) can be used to proportionally reduce the height. Other resizing algorithms including machine learning models are within the scope of the present disclosure. In an example implementation, the multiple images 205 can be compressed during the alignment and merging operations. For example, the multiple images 205 can be downsampled during the alignment and merging operations.

[0032] The image processing module 210 can be configured to align and merge multiple images 205. Aligning and merging multiple images 205 can denoise multiple images 205. Multiple images 205 can be aligned using a Gaussian pyramid technique that moves from a coarse alignment to a finer alignment. Each level of the pyramid can be downsampled. Aligning multiple images 205 can include aligning corresponding pixels of multiple images 205 (for example, pixels representing the same point are shifted to overlap each other). Aligning multiple images 205 can prevent blurring or ghosting due to motion. Merging multiple images 205 can generate an image, and reduce the noise associated with the resulting image (for example, the first image 215-1) relative to the reference image by including similar information from other aligned images. The merge can include a variation of using a Wiener filter for each tile (for example, a portion of an image) across the time dimension. The merge can include generating tiles (portions of images) in each of the multiple images, and using a two-dimensional (2D) DFT across corresponding tiles, thereby generating a spatial frequency domain image merge. The merging may improve color accuracy in darker areas of the resulting image (eg, first image 215 - 1 ).

[0033] The wearable device 110 and the companion device 125 may be communicatively coupled using a wired (e.g., Ethernet) and / or wireless (e.g., Bluetooth) standard. Thus, the first image 215-1 and the second image 220-1 may be transmitted from the wearable device 110 to the companion device 125. Figure 2 , the first image 215-2 and the second image 220-2 represent images that are the same as the first image 215-1 and the second image 220-1, respectively, except that the first image 215-2 and the second image 220-2 are stored in a memory (e.g., a cache) of the companion device 125. The companion device 125 can be configured to generate an HDR image from a burst of images captured by a camera (e.g., an image sensor) of the companion device 125. Thus, the example implementation can use elements of an HDR pipeline associated with the companion device 125. The companion device 125 can be configured to control the wearable device 110 (e.g., transmit, signal, instruction and / or data) to initiate the capture of an HDR image. For example, the companion device 125 can be configured to control the wearable device 110 to trigger the capture of multiple images 205 (e.g., trigger a burst of images). Thus, the wearable device 110 can operate as an input or peripheral device (e.g., image and / or camera input) of the companion device 125.

[0034] The image processing module 230 may be configured to perform color and / or tone mapping (or tuning) on ​​the first image 215-2. Color and / or tone mapping may be an optional operation. Color and / or tone mapping may be configured to map pixels to different color (e.g., RGB) values ​​based on the color and / or tone associated with the pixel and / or surrounding pixels. Color and / or tone mapping may be configured to generate an image that looks more natural. Color and / or tone mapping may be configured to reduce the dynamic range or contrast ratio of the entire image while retaining local contrast. The image processing module 230 may be configured to perform other image finishing operations. For example, the image processing module 230 may be configured to perform lens shading correction, white balance, demosaicing, chromatic aberration correction, sharpening, and / or others.

[0035] The image merging module 235 can be configured to generate an HDR image 240 based on the first image 215-2 and the second image 220-2. The image merging module 235 can be configured to use a machine learning model to merge the first image 215-2 with the second image 220-2. The machine learning model can be a deep learning architecture (e.g., U-Net, CNN, and / or other). The machine learning model can be trained to generate an image with a relatively high resolution (e.g., 12 megapixels) associated with the second image 220-2 and post-processing (e.g., denoising) of the first image 215-2. In other words, the machine learning model can be trained to generate an HDR image based on the first image 215-2 and the second image 220-2.

[0036] The machine learning model can be a machine learning encoder-decoder architecture (e.g., U-Net, CNN, and / or other), wherein an input image is input to an encoder and an output image is output from a decoder. The difference between the input image and the output image can be based on the training of the machine learning encoder-decoder architecture. The image merging module 235 can be configured to generate an HDR image 240 based on the first image 215-2 and the second image 220-2. Therefore, the machine learning encoder-decoder architecture can be used to merge the first image 215-2 and the second image 220-2 to generate the HDR image 240, wherein the first image 215-2 and the second image 220-2 can be input to the encoder, and the HDR image 240 can be output as a decoder.

[0037] As discussed above, the first image 215-2 has a first resolution and the second image 220-2 has a second resolution, wherein the first resolution (e.g., 3 megapixels) is less than the second resolution (e.g., 12 megapixels). Therefore, the encoder of the machine learning encoder-decoder architecture can be configured to take two images of different sizes as input. The first element of the encoder can be a convolution element that takes a first number of channels as input and a second number of channels (typically less than the first number of channels) as output. The number of channels can be based on the number of pixels. For example, each pixel can have channels associated with the red, green, and blue colors of the pixel (other representations of pixels are also within the scope of the present disclosure). In an example implementation, the first element of the encoder (e.g., a convolution) can include a sufficient number of input channels for the number of pixels associated with the first image 215-2 and the second image 220-2.

[0038] In an example implementation, the encoder may include two encoders, and the outputs of the two encoders may be added together (e.g., pixel-by-pixel or channel-by-channel summation) as input to the decoder. The decoder may include a sufficient number of input channels for the added channels of the two encoders. In an example implementation, one of the two encoders may include a sufficient number of input channels for the number of pixels associated with the first image 215-2, and the other encoder may include a sufficient number of input channels for the number of pixels associated with the second image 220-2. As discussed above, the first image 215-2 has a first resolution, and the second image 220-2 has a second resolution, wherein the first resolution (e.g., 3 megapixels) is less than the second resolution (e.g., 12 megapixels). Therefore, the two encoders may have different numbers of input channels. Alternatively, the two encoders may have the same number of input channels, and the input of the encoder associated with the first image 215-2 may be configured to zero-fill the input channels that have no associated pixel inputs to the channel.

[0039] At the location where the two encoder outputs are added together, the number of channels may be equal in number, or unequal in number. After the outputs are added together, the result (e.g., the added channels) can be directly input to the decoder. Alternatively, the result can be operated by an additional encoder. For example, the result can be operated by another convolution. An example implementation has been discussed, as if the outputs of the two encoders were added together. However, in an example implementation, an element configured to add a channel can be at an intermediate step of one of the encoders (e.g., between two convolutions). Therefore, the output of an encoder (e.g., an encoder associated with the first image 215-2) can be an input to another encoder (e.g., an encoder associated with the first image 215-2) where the channels can be added.

[0040] Training a machine learning model can include using HDR true value images. For example, training images representing a first image (e.g., a low-resolution, image-processed image) and a second image (e.g., a high-resolution, original image) can be input into a machine learning model, and the output (e.g., predicted) image can be compared to the HDR true value image. A loss function can be used to evaluate the comparison. The machine learning model can include weights. The weights can be modified or learned during the training operation. For example, if the loss function evaluation produces a loss greater than a threshold, the weights can be modified and training continues. If the loss function evaluation produces a loss less than or equal to a threshold, the training ends. Training can include using supervised learning techniques.

[0041] Continuing with the machine learning encoder-decoder architecture with two encoder examples, training can include inputting a first training image (e.g., a low-resolution, image-processed image) into an encoder and a second training image (e.g., a high-resolution, original image) into another encoder. The first training image and the second training image can be encoded and the associated channels added together. The added channels can be decoded to predict an image (e.g., an HDR image). The predicted image can be compared to an HDR true image. A loss function can be used to evaluate the comparison and the weights associated with the encoder, and the decoder can be modified based on the loss function evaluation. If the loss function evaluation produces a loss less than or equal to a threshold, the training of the machine learning encoder-decoder architecture with two encoders ends. In an example implementation, a machine learning encoder-decoder architecture with two encoders can be trained using multiple first training images, second training images, and HDR true value image combinations.

[0042] Figure 3 A method of operating a wearable device to generate an HDR image according to an example implementation is shown. Figure 3 As shown, in step S305, a plurality of images having a first number of pixels are captured. All of the plurality of images may have the same first number of pixels. For example, a plurality of images may be captured as a burst of images of a wearable device. The burst of images may be a plurality of original (e.g., unprocessed, uncompressed, low-quality, underexposed, noisy, and / or other) images. All of the plurality of images may have a first resolution. The first resolution may be a relatively high resolution (e.g., 12 megapixels). The first resolution may be a variable setting associated with the camera. However, the setting may be immutable during the capture of the burst of images. Therefore, each image of the burst of images may have the same resolution (e.g., number of pixels).

[0043] In step S310, an image having a second number of pixels is generated as a first image based on the multiple images. The first image may be generated based on all images in the multiple images. Generating the image may include performing some image processing on the multiple images. Generating the image may include aligning and merging the multiple images. Aligning and merging the multiple images may denoise the multiple images. The multiple images may be aligned using a Gaussian pyramid technique that moves from a coarse alignment to a finer alignment. Each level of the pyramid may be downsampled. Aligning the multiple images may prevent blurring or ghosting due to motion.

[0044] Merging the multiple images can generate an image, and reducing the noise associated with the resulting image relative to the reference image by including similar information from other aligned images. The merge can include a variation of using a Wiener filter for each tile (e.g., a portion of an image) across a time dimension. The merge can improve the color accuracy in the darker areas of the resulting image. The multiple images can be downsampled during the alignment and merging operations. The multiple images can be compressed before, during and / or after the alignment and merging operations. Downsampling and / or compressing the multiple images and / or the image can generate an image with a second number of pixels. The second number of pixels can be less than the first number of pixels. Therefore, the image with the second number of pixels can have a lower resolution (e.g., 3 megapixels) than the multiple images.

[0045] In step S315, one of the multiple images is selected as the second image. The second image may have the same number of pixels as each of the multiple images. In other words, the second image may have a relatively high resolution (e.g., 12 megapixels). The second image may be an unprocessed image or a raw image. The second image 120 may be selected based on a setting of the wearable device. For example, a preconfigured setting or a user setting may be stored in a memory. The setting may indicate, for example, that the first image, the last image, the middle image, and / or the like in a time sequence of multiple images is to be selected. Alternatively, the setting may include a randomly selected image in a time sequence of multiple images. Alternatively, the setting may include determining the best image (e.g., a certain quality threshold) and selecting the best image in the time sequence of the multiple images.

[0046] In step S320, the first image and the second image are transmitted to the supporting device. For example, the wearable device and the supporting device can be communicatively coupled using a wired (e.g., Ethernet) and / or wireless (e.g., Bluetooth) standard. Therefore, the first image and the second image can be transmitted from the wearable device to the supporting device.

[0047] Figure 4 A method of operating a companion device to generate an HDR image according to an example implementation is shown. Figure 4 As shown, in step S405, a first image and a second image are received from the wearable device. The first image may have a relatively low resolution (e.g., 3 megapixels), and the second image may have a relatively high resolution (e.g., 12 megapixels). The first image may be an image processed, and the second image may be an unprocessed (e.g., unprocessed) image or an original image.

[0048] In step S410, the first image is processed. Processing the first image may include color and / or tone mapping (or tuning) the first image. Color and / or tone mapping may be an optional operation. Color and / or tone mapping may be configured to map the pixel to different color (e.g., RGB) values ​​based on the color and / or tone associated with the pixel and / or surrounding pixels. Color and / or tone mapping may be configured to generate an image that looks more natural. Color and / or tone mapping may be configured to reduce the dynamic range or contrast ratio of the entire image while retaining local contrast. Processing the first image may include other image finishing operations. For example, processing the first image may include performing lens shading correction, white balance, demosaicing, chromatic aberration correction, sharpening, and / or others.

[0049] In step S415, the first image is merged with the second image to generate a high dynamic range (HDR) image. For example, merging the first image with the second image may include using a machine learning model to merge the first image with the second image. The machine learning model may be a deep learning architecture (e.g., U-Net). The machine learning model may be trained to generate an image with a relatively high resolution (e.g., 12 megapixels) associated with the second image and post-processing (e.g., denoising) of the first image. In other words, the machine learning model may be trained to generate an HDR image based on the first image and the second image.

[0050] Figure 5A A wearable device configured to generate an HDR image according to an example implementation is shown. Figure 5A In the example of the wearable device 110 (e.g., smart glasses) may include a computing system or at least one computing device, and should be understood to represent substantially any computing device configured to perform the techniques described herein. Thus, the device may be understood to include various components that may be used to implement the techniques described herein or different or future versions thereof. For example, the system may include a processor 505, a memory 510 (e.g., a non-transitory computer-readable memory), and a camera 515. The processor 505 and the memory 510 may be coupled (e.g., communicatively coupled) via a bus.

[0051] The processor 505 may be used to execute instructions stored on at least one memory 510. Thus, the processor 505 may implement various features and functions described herein or additional or alternative features and functions. The processor 505 and at least one memory 510 may be used for various other purposes. For example, at least one memory 510 may represent examples of various types of memory and related hardware and software that may be used to implement any of the modules described herein.

[0052] At least one memory 510 may be configured to store data and / or information associated with the device. At least one memory 510 may be a shared resource. Therefore, at least one memory 510 may be configured to store data and / or information associated with other elements (e.g., image / video processing or wired / wireless communication) within a larger system. The processor 505 and at least one memory 510 may be used together to implement the techniques described herein. Therefore, the techniques described herein may be implemented as code segments (e.g., software) stored on the memory 510 and executed by the processor 505. Therefore, the memory 510 may include an image processing module 210. As discussed above, the image processing module 210 may be configured to generate a first image and a second image based on a plurality of images. Generating a first image may include performing some image processing on the plurality of images. Generating a second image may include selecting one of the plurality of images. The second image 120 may be selected based on the settings of the wearable device. For example, preconfigured settings or user settings may be stored in the memory. The settings may indicate, for example, that the first image, the last image, the intermediate image, and / or the like in the time sequence of the plurality of images is to be selected.

[0053] Figure 5B A supporting apparatus configured to generate an HDR image according to an example implementation is shown. Figure 5B In the example of , the supporting device (e.g., a mobile phone) may include a computing system or at least one computing device, and should be understood to represent almost any computing device configured to perform the techniques described herein. Therefore, the device can be understood to include various components that can be used to implement the techniques described herein or different versions or future versions thereof. For example, the system may include a processor 520 and a memory 525 (e.g., a non-transitory computer-readable memory). The processor 520 and the memory 525 can be coupled (e.g., communicatively coupled) via a bus.

[0054] The processor 520 may be used to execute instructions stored on at least one memory 525. Thus, the processor 520 may implement various features and functions described herein or additional or alternative features and functions. The processor 520 and at least one memory 525 may be used for various other purposes. For example, at least one memory 525 may represent examples of various types of memory and related hardware and software that may be used to implement any of the modules described herein.

[0055] At least one memory 525 may be configured to store data and / or information associated with the device. At least one memory 525 may be a shared resource. Therefore, at least one memory 525 may be configured to store data and / or information associated with other elements (e.g., image / video processing or wired / wireless communication) within a larger system. The processor 520 and at least one memory 525 may be used together to implement the techniques described herein. Therefore, the techniques described herein may be implemented as code segments (e.g., software) stored on the memory 525 and executed by the processor 520. Therefore, the memory 525 may include an image processing module 230 and a machine learning module 235. As discussed above, the image processing module 230 may be configured to perform color and / or tone mapping (or tuning) on ​​the image. The image processing module 230 may be configured to perform other image sorting operations. As discussed above, the image merging module 235 may be configured to generate an HDR image based on a first image (e.g., a low-resolution image) and a second image (e.g., a high-resolution image).

[0056] Figure 6 A wearable device according to an example implementation is shown. Figure 6 As shown, the wearable device 600 includes a frame 605, a frame 610, a center frame support 615, a lens element 620, a lens element 625, an extended side leg 630, an extended side leg 635, an image capture device 640 (e.g., a camera), an on-board computing system 645, a speaker 650, and a microphone 655.

[0057] Each of the frame elements 605, 610, and 615 and the extended side legs 630, 635 can be formed of a solid structure of plastic and / or metal, or can be formed of a hollow structure of similar materials to allow wiring and component interconnections to be routed through the wearable device 600 from the inside. Other materials are also possible. At least one of the lens elements 620, 625 can be formed of any material that can appropriately display the projected image or graphics. Each of the lens elements 620, 625 can also be transparent enough to allow a user to see through the lens element. Combining these two features of the lens element can facilitate an augmented reality or head-up display in which the projected image or graphics are superimposed on the real-world view as perceived by the user through the lens element.

[0058] The central frame support 615 and the extended side legs 630, 635 are configured to secure the wearable device 600 to the user's face via the user's nose and ears, respectively. The extended side legs 630, 635 can each be an extension extending away from the frame 605, 610, and can be positioned behind the user's ears to secure the wearable device 600 to the user. The extended side legs 630, 635 can further secure the wearable device 600 to the user by extending around the back portion of the user's head. Additionally or alternatively, for example, the wearable device 600 can be connected to or attached within a head-mounted helmet structure. Other configurations for wearable computing devices are also possible.

[0059] The on-device computing system 645 is shown as being positioned on the extended side leg 630 of the wearable device 600; however, the on-device computing system 645 can be disposed on other portions of the wearable device 600, or can be positioned remote from the wearable device 600 (e.g., the on-device computing system 645 can be wired or wirelessly connected to the wearable device 600). For example, the on-device computing system 645 can include a processor and a memory. The on-device computing system 645 can be configured to receive and analyze data from the image capture device 640 (and possibly from other sensing devices) and generate images for output by the lens elements 620, 625.

[0060] The image capture device 640 can be, for example, a camera configured to capture still images and / or capture video. In the illustrated configuration, the image capture device 640 is positioned on the extended side leg 630 of the wearable device 600; however, the image capture device 640 can also be disposed on other portions of the wearable device 600. The image capture device 640 can be configured to capture images at various resolutions or at different frame rates. Many image capture devices with a small form factor, such as cameras used in mobile phones or webcams, can be incorporated into the example of the wearable device 600.

[0061] One image capture device 640 is shown. However, more image capture devices may be used, and each image capture device may be configured to capture the same view, or to capture a different view. For example, the image capture device 640 may face forward to capture at least a portion of the real-world view perceived by the user. This forward-facing image captured by the image capture device 640 may then be used to generate an augmented reality in which a computer-generated image appears to interact with or overlay the real-world view perceived by the user.

[0062] An example implementation may include a non-transitory computer-readable storage medium including instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to perform any of the methods described above. An example implementation may include a device including means for performing any of the methods described above. An example implementation may include a device including at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, with the at least one processor, cause the device to at least perform any of the methods described above.

[0063] The split computing architecture can be an architecture that moves the application runtime environment to a remote computing endpoint or supporting device (for simplicity, usually referred to as a mobile device or mobile phone) such as a mobile phone, server, cloud, desktop computer, etc. In some implementations, data sources such as IMU and camera sensors can be streamed from a wearable device to a supporting device. In some implementations, display content can be streamed back to the wearable device from a supporting device. In some implementations, data streams, video / audio streams, bidirectional control streams, and / or other can be transmitted between a wearable device and a supporting device. In some implementations, because part of the calculation and rendering may not occur on the wearable device itself, the split computing architecture can allow the use of a low-power MCU-based system. In some implementations, this can allow control power and ID to be maintained, design constraints to be met, etc. In the case of codecs and networking, it is possible to maintain the required networking bandwidth in a low-power manner. In some implementations, a wearable device can be connected to more than one supporting device at a given time. In some implementations, different supporting devices can provide different services. In some implementations, as low-latency, high-bandwidth 5G connections become mainstream, the companion device can exist in the cloud. In some implementations, the wearable device can communicate with the companion device through a protocol to be clearly defined. The architecture can be platform-independent.

[0064] Figure 7An example of a computer device 700 and a mobile computer device 750 that can be used with the techniques described herein (e.g., to implement a wearable device 110 and a companion device 125) is shown. The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 connected to the memory 704 and a high-speed expansion port 710, and a low-speed interface 712 connected to a low-speed bus 714 and the storage device 706. Each of the components 702, 704, 706, 708, 710, and 712 is interconnected using various buses and can be mounted on a common motherboard or otherwise installed where appropriate. The processor 702 can process instructions for execution within the computing device 700, including instructions stored in the memory 704 or on the storage device 706, to display graphical information of a GUI on an external input / output device (such as a display 716 coupled to the high-speed interface 708). In other implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple types of memories where appropriate. Furthermore, multiple computing devices 700 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0065] The memory 704 stores information within the computing device 700. In one implementation, the memory 704 is one or more volatile memory units. In another implementation, the memory 704 is one or more non-volatile memory units. The memory 704 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0066] The storage device 706 can provide mass storage for the computing device 700. In one implementation, the storage device 706 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory or another similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 704, the storage device 706, or a memory on the processor 702.

[0067] The high-speed controller 708 manages bandwidth-intensive operations of the computing device 700, while the low-speed controller 712 manages less bandwidth-intensive operations. This allocation of functions is merely an example. In one implementation, the high-speed controller 708 is coupled to the memory 704, the display 716 (e.g., through a graphics processor or accelerator), and to the high-speed expansion port 710 that can accept various expansion cards (not shown). In an implementation, the low-speed controller 712 is coupled to the storage device 706 and the low-speed expansion port 714. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device (such as a switch or router), for example, through a network adapter.

[0068] The computing device 700 can be implemented in a variety of different forms, as shown. For example, it can be implemented as a standard server 720, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 724. In addition, it can also be implemented in a personal computer (such as a laptop computer 722). Alternatively, components from the computing device 700 can be combined with other components in a mobile device (not shown) (such as device 750). Each of such devices can contain one or more of the computing devices 700, 750, and the entire system can be composed of multiple computing devices 700, 750 that communicate with each other.

[0069] Computing device 750 includes processor 752, memory 764, input / output devices such as display 754, communication interface 766 and transceiver 768, among other components. Device 750 may also be provided with storage devices such as microdrives or other devices to provide additional storage. Each of components 750, 752, 764, 754, 766 and 768 are interconnected using various buses, and several of these components may be mounted on a common motherboard or otherwise where appropriate.

[0070] Processor 752 can execute instructions within computing device 750, including instructions stored in memory 764. The processor can be implemented as a chipset including a separate chip with multiple analog and digital processors. For example, the processor can provide coordination of other components of device 750, such as control of a user interface, applications executed by device 750, and wireless communications performed by device 750.

[0071] The processor 752 can communicate with the user through the control interface 758 and the display interface 756 coupled to the display 754. The display 754 can be, for example, a TFT LCD (thin film transistor liquid crystal display) and an LED (light emitting diode) or OLED (organic light emitting diode) display or other appropriate display technology. The display interface 756 may include an appropriate circuit system for driving the display 754 to present graphical information and other information to the user. The control interface 758 can receive commands from the user and convert them for submission to the processor 752. In addition, an external interface 762 in communication with the processor 752 can be provided to enable near-field communication of the device 750 with other devices. In some implementations, the external interface 762 can provide, for example, wired communication, or in other implementations, wireless communication, and multiple interfaces can also be used.

[0072] The memory 764 stores information within the computing device 750. The memory 764 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 774 may also be provided and connected to the device 750 via an expansion interface 772, which may include, for example, a SIMM (single in-line memory module) card interface. Such an expansion memory 774 may provide additional storage space for the device 750, or may also store applications or other information of the device 750. Specifically, the expansion memory 774 may include instructions for executing or supplementing the processes described above, and may also include security information. Thus, for example, the expansion memory 774 may be provided as a security module for the device 750, and may be programmed with instructions that permit secure use of the device 750. In addition, a security application may be provided via a SIMM card together with additional information, such as placing identification information on the SIMM card in an unbreakable manner.

[0073] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or machine-readable medium, such as memory 764, expansion memory 774, or memory on processor 752, which may be received, for example, via transceiver 768 or external interface 762.

[0074] The device 750 can communicate wirelessly via a communication interface 766, which may include a digital signal processing circuit system when necessary. The communication interface 766 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. Such communication can occur, for example, via a radio frequency transceiver 768. In addition, short-range communication can occur, such as using Bluetooth, Wi-Fi or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 770 can provide additional navigation and positioning related wireless data to the device 750, which can be used by applications running on the device 750 where appropriate.

[0075] Device 750 may also communicate audibly using audio codec 760, which may receive verbal information from a user and convert it into usable digital information. Audio codec 760 may also generate audible sounds for the user, such as through a speaker (e.g., in an earpiece of device 750). Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 750.

[0076] The computing device 750 can be implemented in many different forms, as shown in the figure. For example, it can be implemented as a cellular phone 780. It can also be implemented as part of a smart phone 782, a personal digital assistant or other similar mobile device.

[0077] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0078] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium," "computer-readable medium," refer to any computer program product, apparatus, and / or device (e.g., a disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0079] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (LED (light emitting diode), or OLED (organic LED), or LCD (liquid crystal display) monitor / screen) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, voice, or tactile input.

[0080] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0081] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0082] In some implementations, the computing device depicted in the figure may include a sensor that interfaces with the AR head-mounted device / HMD device 790 to generate an augmented environment for viewing inserted content in the physical space. For example, one or more sensors included on the computing device 750 depicted in the figure or other computing devices may provide input to the AR head-mounted device 790, or generally speaking, to the AR space. Sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 750 may use sensors to determine the absolute position and / or detected rotation of the computing device in the AR space, and then the absolute position and / or the detected rotation may be used as input to the AR space. For example, the computing device 750 may be incorporated into the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, etc. The positioning of the computing device / virtual object by the user when incorporated into the AR space may allow the user to position the computing device so as to view the virtual object in some way in the AR space. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device as if it were an actual laser pointer. The user can move the computing device left to right, up and down, in circles, etc., and use the device in a manner similar to using a laser pointer. In some implementations, the user can aim at a target using the virtual laser pointer.

[0083] In some implementations, one or more input devices included on or connected to the computing device 750 can be used as input to the AR space. The input device can include, but is not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, headphones or earbuds with input functionality, a game controller, or other connectable input device. When the computing device 750 is incorporated into the AR space, a user interacting with the input device included on the computing device can cause specific actions to occur in the AR space.

[0084] In some implementations, the touch screen of the computing device 750 can be rendered as a touch pad in the AR space. The user can interact with the touch screen of the computing device 750. For example, in the AR head mounted device 790, the interaction is rendered as movement on the rendered touch pad in the AR space. The rendered movement can control the virtual object in the AR space.

[0085] In some implementations, one or more output devices included on the computing device 750 can provide output and / or feedback to a user of the AR head mounted device 790 in the AR space. The output and feedback can be visual, tactile, or auditory. The output and / or feedback can include, but are not limited to, vibration, turning on and off or blinking and / or flashing one or more lights or flashes, sounding an alarm, playing a chime, playing a song, and playing an audio file. The output device can include, but is not limited to, a vibration motor, a vibration coil, a piezoelectric device, an electrostatic device, a light emitting diode (LED), a flash, and a speaker.

[0086] In some implementations, computing device 750 may appear as another object in a computer-generated 3D environment. Interaction by a user with computing device 750 (e.g., rotating, shaking, touching a touch screen, sliding a finger across a touch screen) may be interpreted as interaction with an object in AR space. In the example of a laser pointer in AR space, computing device 750 appears as a virtual laser pointer in a computer-generated 3D environment. As the user manipulates computing device 750, the user in AR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 750 in the AR environment on computing device 750 or AR head-mounted device 790. User interaction with the computing device can be converted into interaction with a user interface generated for a controllable device in the AR environment.

[0087] In some implementations, computing device 750 may include a touch screen. For example, a user may interact with the touch screen to interact with a user interface of a controllable device. For example, the touch screen may include user interface elements, such as sliders, that may control properties of the controllable device.

[0088] Computing device 700 is intended to represent various forms of digital computers and devices, including but not limited to laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 750 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be examples only, and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0089] An example implementation may be embodied as a non-transitory computer-readable storage medium including instructions stored thereon, which when executed by at least one processor are configured to cause a computing system to perform multiple operations and / or steps based on the above implementation. An example implementation may be embodied as a device including components for performing multiple operations and / or steps based on the above implementation. An example implementation may be embodied as a device including at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to cause the device to perform multiple operations and / or steps based at least on the above implementation together with the at least one processor.

[0090] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the specification.

[0091] In addition, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve the desired results. In addition, other steps can be provided, or steps can be deleted from the described processes, and other components can be added to or removed from the described systems. Therefore, other embodiments are within the scope of the appended claims.

[0092] In addition to the above, controls may be provided to allow the user to make choices about whether and when the systems, programs, or features described herein may be able to collect user information (e.g., information about the user's social network, social behavior, or activities, occupation, user preferences, or the user's current location) and whether to send content or communications from a server to the user. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location for which location information is obtained may be generalized (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user may control what information is collected about the user, how the information is used, and what information is provided to the user.

[0093] Although certain features of the described implementations have been shown as described herein, those skilled in the art will now expect many modifications, substitutions, variations and equivalents. Therefore, it should be understood that the appended claims are intended to cover all these modifications and variations that fall within the scope of the implementations. It should be understood that they are presented only in an exemplary and non-restrictive manner, and various changes in form and detail may be made. Except for mutually exclusive combinations, any part of the devices and / or methods described herein may be combined in any combination. The implementations described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations described.

[0094] Although the example embodiments may include various modifications and alternative forms, embodiments thereof are shown by way of example in the drawings and will be described in detail herein. However, it should be understood that it is not intended to limit the example embodiments to the particular forms disclosed, but rather, the example embodiments will cover all modifications, equivalents, and alternatives falling within the scope of the claims. Like numbers refer to like elements in the description of the drawings.

[0095] Some of the above example embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations as sequential processes, many operations may be performed in parallel, concurrently, or simultaneously. In addition, the order of the operations may be rearranged. Processes may terminate when their operations are completed, but may also have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc.

[0096] The methods discussed above (some of which are illustrated by flowcharts) may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the necessary tasks may be stored in a machine or computer readable medium, such as a storage medium. A processor may perform the necessary tasks.

[0097] The specific structural and functional details disclosed herein are merely representative for describing example embodiments. Example embodiments may, however, be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.

[0098] It should be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the exemplary embodiment, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the terms and / or include any and all combinations of one or more of the associated listed items.

[0099] It should be understood that when an element is referred to as being connected or coupled to another element, the element may be directly connected or coupled to the other element, or there may be intervening elements. Conversely, when an element is referred to as being directly connected or coupled to another element, there are no intervening elements. Other words used to describe the relationship between elements should be interpreted in a similar manner (e.g., between versus directly between, adjacent versus directly adjacent, etc.).

[0100] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an", "a kind of", and "the" are also intended to include the plural forms. It should be further understood that the terms "include", "comprise", and / or "contain" when used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0101] It should also be noted that in some alternative implementations, the functions / behaviors noted may not occur in the order noted in the figures. For example, two figures shown in succession may actually be executed simultaneously or may sometimes be executed in the reverse order, depending on the functionality / behaviors involved.

[0102] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It should be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless explicitly defined as such herein.

[0103] Portions of the above exemplary embodiments and corresponding detailed descriptions are presented in terms of software or algorithms and symbolic representations of operations on data bits in computer memory. These descriptions and representations are descriptions and representations that are used by those of ordinary skill in the art to effectively convey the substance of their work to other persons of ordinary skill in the art. As the term is used herein, and as generally used, an algorithm is considered to be a self-consistent sequence of steps that produces a desired result. These steps are steps that require physical manipulation of physical quantities. Although not necessary, these quantities are typically in the form of optical, electrical, or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated. It has been demonstrated that, primarily for common usage reasons, these signals can sometimes be conveniently referred to as bits, values, elements, symbols, characters, terms, numbers, etc.

[0104] In the above illustrative embodiments, references to symbolic representations of actions and operations that can be implemented as program modules or functional processes (e.g., in the form of flowcharts) include routines, programs, objects, components, data structures, etc., which perform specific tasks or implement specific abstract data types and can be described and / or implemented using existing hardware at existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate arrays (FPGAs), computers, etc.

[0105] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically noted or as is apparent from the discussion, terms such as processing or computing or calculating or determining display refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical electronic quantities within the computer system's registers and memories and transforms that data into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0106] It is also noted that the software-implemented aspects of the example embodiments are typically encoded on some form of non-transitory program storage medium or implemented on some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or hard disk) or optical (e.g., a compact disk read-only memory or CD ROM), and may be read-only or random access. Similarly, the transmission medium may be twisted pair, coaxial cable, optical fiber, or some other suitable transmission medium known in the art. The example embodiments are not limited by these aspects of any given implementation.

[0107] Finally, it should also be noted that although the appended claims set forth specific combinations of features described herein, the scope of the present disclosure is not limited to the specific combinations claimed below, but extends to cover any combination of features or embodiments disclosed herein, regardless of whether the specific combination is specifically listed in the appended claims.

Claims

1. A non-transitory computer-readable storage medium comprising instructions stored thereon, the instructions being configured to cause a wearable device, when executed by at least one processor: capturing, by the wearable device, a plurality of images having a first number of pixels; processing, by the wearable device, the plurality of images to generate a first image having a second number of pixels less than the first number of pixels; selecting, by the wearable device based on a setting of the wearable device, one of the plurality of images having the first number of pixels as a second image; and The wearable device transmits the first image and the second image to a companion device to generate a high dynamic range (HDR) image.

2. The non-transitory computer-readable storage medium of claim 1, wherein the instructions further cause the wearable device to: The wearable device controls the supporting device to generate the HDR image based on the first image and the second image.

3. The non-transitory computer-readable storage medium of claim 1 or 2, wherein: The capturing of the plurality of images is an image burst comprising capturing a series of images in response to a trigger.

4. The non-transitory computer-readable storage medium of any one of the preceding claims, wherein: Processing the plurality of images to generate the first image includes compressing at least one of the first image and each image in the plurality of images.

5. The non-transitory computer-readable storage medium of any one of claims 1 to 3, wherein: Processing the plurality of images to generate the first image includes downsampling at least one of the first image and each image in the plurality of images.

6. The non-transitory computer-readable storage medium of any one of the preceding claims, the instructions further causing the wearable device to: receiving a signal from the supporting device, wherein: The capturing of the plurality of images is in response to receiving the signal.

7. The non-transitory computer-readable storage medium of any one of the preceding claims, wherein: Processing the plurality of images to generate the first image comprises: reducing the number of pixels of each of the plurality of images to the second number of pixels; aligning corresponding pixels of the plurality of images; and The multiple images are merged to generate the first image.

8. The non-transitory computer-readable storage medium of any one of the preceding claims, wherein: Processing the plurality of images to generate the first image includes tone mapping and color tuning the image having the second number of pixels.

9. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising instructions stored thereon, the instructions being configured to cause a companion device, when executed by at least one processor: Receiving, by the companion device, a first image having a first resolution and a second image having a second resolution from the wearable device, the first resolution being smaller than the second resolution, the first image and the second image being generated based on an image burst comprising capturing a series of images in response to a trigger; The first image is processed by the supporting device; and The supporting device merges the first image with the second image to generate a high dynamic range (HDR) image.

10. The non-transitory computer-readable storage medium of claim 9, wherein: The merging of the first image and the second image comprises: inputting the first image and the second image into a machine learning model; and The HDR image is generated based on the first image and the second image using the machine learning model.

11. The non-transitory computer-readable storage medium of claim 10, wherein: The machine learning model is trained using HDR ground truth images.

12. The non-transitory computer-readable storage medium of any one of claims 9 to 11, wherein the instruction further causes the companion device to: The wearable device is controlled by the companion device to initiate the generation of the HDR image based on the first image and the second image.

13. The non-transitory computer-readable storage medium of any one of claims 9 to 12, wherein the instruction further causes the companion device to: The signal is transmitted to the wearable device, wherein: The signal causes the wearable device to trigger the image burst.

14. A system comprising a wearable device and a supporting device, the system comprising: at least one processor; as well as at least one memory, the at least one memory comprising computer program code; The at least one memory and the computer program code are configured to, with the at least one processor, cause the system to: capturing, by the wearable device, a plurality of images each having a first resolution; processing, by the wearable device, the plurality of images to generate a first image having a second resolution, the second resolution being less than the first resolution; selecting, by the wearable device based on a setting of the wearable device, a second image from the plurality of images having the first resolution; The wearable device transmits the first image and the second image to the supporting device; Processing the first image by the supporting device; and The supporting device merges the processed first image with the second image to generate a high dynamic range (HDR) image.

15. The system of claim 14, wherein: Processing the plurality of images to generate the first image comprises: reducing the resolution of each of the plurality of images to the second resolution; aligning corresponding pixels of the plurality of images; and The multiple images are merged to generate the first image.

16. The system of claim 14 or 15, wherein: The processing of the first image includes tone mapping and color tuning the first image.

17. A system as claimed in any one of claims 14 to 16, wherein: The merging of the first image and the second image comprises: inputting the processed first image and the second image into a machine learning model; and The HDR image is generated based on the processed first image and the second image using the machine learning model.

18. The system of claim 17, wherein: The machine learning model is trained using HDR ground truth images.

19. The system of any one of claims 14 to 18, further comprising: The wearable device is controlled by the companion device to initiate the generation of the HDR image based on the first image and the second image.

20. The system of any one of claims 14 to 19, further comprising: A signal is transmitted from the companion device to the wearable device, the signal causing the wearable device to trigger the capture of the multiple images.

21. The system of any one of claims 14 to 20, further comprising: The wearable device controls the supporting device to generate the HDR image based on the first image and the second image.

22. The system of any one of claims 14 to 21, wherein: The capturing of the plurality of images is an image burst comprising capturing a series of images in response to a trigger.

23. A method comprising: capturing, by the wearable device, a plurality of images having a first number of pixels; processing, by the wearable device, the plurality of images to generate a first image having a second number of pixels less than the first number of pixels; selecting, by the wearable device based on a setting of the wearable device, one of the plurality of images having the first number of pixels as a second image; as well as The wearable device transmits the first image and the second image to a companion device to generate a high dynamic range (HDR) image.

24. A wearable device comprising: at least one processor; as well as at least one memory, the at least one memory comprising computer program code; The at least one memory and the computer program code are configured to, with the at least one processor, cause the wearable device to: capturing, by the wearable device, a plurality of images having a first number of pixels; processing, by the wearable device, the plurality of images to generate a first image having a second number of pixels less than the first number of pixels; selecting, by the wearable device based on a setting of the wearable device, one of the plurality of images having the first number of pixels as a second image; and The wearable device transmits the first image and the second image to a companion device to generate a high dynamic range (HDR) image.

25. A wearable device comprising means for: capturing, by the wearable device, a plurality of images having a first number of pixels; processing, by the wearable device, the plurality of images to generate a first image having a second number of pixels less than the first number of pixels; selecting, by the wearable device based on a setting of the wearable device, one of the plurality of images having the first number of pixels as a second image; as well as The wearable device transmits the first image and the second image to a companion device to generate a high dynamic range (HDR) image.

26. A method comprising: Receiving, by the companion device, a first image having a first resolution and a second image having a second resolution from the wearable device, the first resolution being smaller than the second resolution, the first image and the second image being generated based on an image burst comprising capturing a series of images in response to a trigger; Processing the first image by the supporting device; as well as The supporting device merges the first image with the second image to generate a high dynamic range (HDR) image.

27. A matching device, comprising: at least one processor; as well as at least one memory, the at least one memory comprising computer program code; The at least one memory and the computer program code are configured to, together with the at least one processor, cause the companion device to: Receiving, by the companion device, a first image having a first resolution and a second image having a second resolution from the wearable device, the first resolution being smaller than the second resolution, the first image and the second image being generated based on an image burst comprising capturing a series of images in response to a trigger; The first image is processed by the supporting device; and The supporting device merges the first image with the second image to generate a high dynamic range (HDR) image.

28. A kit comprising components for: Receiving, by the companion device, a first image having a first resolution and a second image having a second resolution from the wearable device, the first resolution being smaller than the second resolution, the first image and the second image being generated based on an image burst comprising capturing a series of images in response to a trigger; Processing the first image by the supporting device; as well as The supporting device merges the first image with the second image to generate a high dynamic range (HDR) image.

Images