Foveal image with adaptive exposure

Abstract

A system and method are disclosed to address the need for adaptive exposure in high dynamic range (HDR) images. The solution can leverage recent advances in the use of virtual reality (VR) headsets and augmented reality (AR) displays equipped with infrared (IR) eye-tracking devices. The gaze vector determined by the eye-tracking device identifies one or more fixation points on the image corresponding to areas where underexposure exists. Exposure around the fixation points can be adaptively corrected using image processing techniques. Using spatially adaptive exposure, the resulting image, a kind of foveal image, can be rendered with sufficient detail on a low dynamic range (LDR) display.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application is a continuation of U.S. Application No. 18 / 316,710, filed on May 12, 2023, and claims the benefit thereof, and this disclosure is hereby incorporated by reference in its entirety into this specification.

Background Art

[0002] Wearable computing devices used to create augmented reality (AR) experiences can include, for example, head - mounted wearable devices, wrist - mounted wearable devices, hand - mounted wearable devices, pendants, etc. Head - mounted wearable devices for extended reality (XR / AR / VR) can include, for example, earphones and head - mounted eyewear such as smart glasses, headsets or goggles. A camera can be disposed in the head - mounted eyewear, and an image can be projected onto the lens of the head - mounted eyewear to provide a head - up display (HUD). The camera attached to the frame of the head - mounted eyewear can include an outward - facing camera and an eyeball / gaze - tracking device. The content displayed on the head - up display can include images and information received from an outward - facing camera, the Internet, or other sensory inputs. Since the eyeball / gaze - tracking device can provide feedback to the AR / VR system to continuously adjust the display, the display is projected onto the area of the lens that the user is looking at. Wrist / hand - mounted accessories can include, for example, smartwatches, smart bracelets, smart rings, etc. Wearable computing devices can include various types of electronic components for computing and both long - range and short - range radio frequency (RF) wireless communication.

Summary of the Invention

[0003] This disclosure describes a method and system for applying spatially - adaptive exposure to digital HDR images.

[0004] In some embodiments, the techniques described herein are methods comprising: identifying a digital image having underexposed or overexposed areas; tracking the eye movements of an observer of the digital image to determine a fixation point; applying a Gaussian mask to the digital image around the fixation point; calculating the pixel intensity for each pixel in the Gaussian mask; increasing the exposure setting of the pixels in the digital image if the pixel intensity is darker than a dark threshold; decreasing the exposure setting of the pixels in the digital image if the pixel intensity is brighter than a bright threshold; and displaying the digital image to a user.

[0005] In some embodiments, the technology described herein relates to a system comprising a display and an outward-facing camera mounted on the display, the outward-facing camera configured to produce a digital image; the system further comprises an eye-tracking device mounted on the display, the eye-tracking device configured to identify selected pixels of the digital image; and the system further comprises a graphics processing unit (GPU) communicably coupled to the outward-facing camera, the eye-tracking device and the display, the GPU configured to adaptively correct the exposure of selected pixels to produce a foveated image for projection onto the display.

[0006] In some embodiments, the technology described herein relates to a headset comprising a wearable display, an outward-facing camera attached to the wearable display, an eye-tracking device attached to the wearable display, and a graphics processing unit (GPU) coupled to the outward-facing camera, the eye-tracking device, and the wearable display, wherein the GPU is configured to receive image data and identified regions of interest, adjust the exposure of portions of the image data corresponding to the identified regions of interest to produce a foveal image, and project the foveal image onto the wearable display.

[0007] The above-mentioned overview, as well as other exemplary purposes and / or advantages of the present disclosure, and the methods by which they are achieved, are further described in the embodiments for carrying out the invention and the accompanying drawings below. [Brief explanation of the drawing]

[0008] [Figure 1] A, B, and C illustrate the application of adaptive exposure to a first example of a digital HDR image according to possible embodiments of the present disclosure. [Figure 2] A, B, C, and D illustrate the application of adaptive exposure to a second example of a digital HDR image according to possible embodiments of the present disclosure. [Figure 3] This is a schematic diagram of a user wearing smart glasses according to a possible embodiment of the present disclosure. [Figure 4] This is a front view of smart glasses according to a possible embodiment of the present disclosure. [Figure 5] This is a rear view of smart glasses showing the arrangement of the IMU according to a possible embodiment of the present disclosure. [Figure 6] A and B illustrate the operation of an eye-tracking device according to possible embodiments of the present disclosure. [Figure 7] A and B illustrate the operation of an eye-tracking device according to possible embodiments of the present disclosure. [Figure 8] This is a block diagram of a system for creating foveal images with adaptive exposure according to possible embodiments of the present disclosure. [Figure 9] This is a data flow diagram corresponding to the system shown in Figure 8, according to a possible embodiment of the present disclosure. [Figure 10] This is a flowchart of a method for creating a foveal image with adaptive exposure according to possible embodiments of the present disclosure. [Figure 11] This is a block diagram of an Internet server system according to possible embodiments of the present disclosure. [Modes for carrying out the invention]

[0009] The components in the drawings are not necessarily to scale relative to each other. Similar reference numbers indicate corresponding parts across several drawings.

[0010] In the physical world, humans perceive images through adaptive exposure. For example, the human eye adapts to see stars in the night sky when there is light nearby. Under photopic vision, or light-adaptive vision, both the rods and cones of the human eye are active with sensitivity to light at approximately 680 lumens per watt. Under scotopic vision, or dark-adaptive vision, only the rods of the human eye are active, resulting in sensitivity to light at 1700 lumens per watt. Thus, the human eye automatically adapts to light levels by including or excluding cones as sensors.

[0011] In the digital world, the lack of automatic adaptation to light levels leads to several problems. Firstly, high dynamic range (HDR) images are stored as 10-bit, 12-bit, or 16-bit data blocks (resolution), while the image is rendered on a low dynamic range (LDR) display with 3x8-bit channels. Consequently, details of the captured image are lost during the process of displaying the image to the user. Secondly, when displaying different parts of the same HDR image, the brightness of the pixels remains uniform. With 8-bit channels, global values ​​are used to change the brightness and contrast of the entire image without adapting to spatial variations in brightness. As a result, a technical problem with HDR images is that parts of the image may be overexposed or underexposed, blurring the details in those areas.

[0012] The systems and methods disclosed herein address the need for adaptive exposure in HDR images. Embodiments can apply adaptive exposure in a spherical panoramic image referred to as a 360-degree HDR “image” or “dome.” Such images can be captured by a 360 camera or derived from a composite 3D scene. The solutions described herein leverage recent advances in the use of VR headsets and AR displays equipped with infrared (IR) eye-tracking devices. Using IR eye tracking, the user’s gaze direction can be detected with high precision to identify a precise area of ​​an image that captures the user’s attention. IR eye tracking involves illuminating the pupil of the eye with infrared or near-infrared light to generate a reflection from the surface of the cornea, which can be recorded by an optical sensor or an IR camera, for example, an eye-tracking device located on the frame of VR / AR glasses. For example, by processing changes in the reflection data at time intervals of 1 / 120 seconds (corresponding to 12 Hz), eye rotation can be determined, and then the user’s gaze direction, i.e., gaze vector, can be determined.

[0013] The line-of-sight vector identifies one or more fixation points on the image corresponding to the region of interest. The region of interest may be an underexposed area. Underexposure refers to an area of ​​the image that is too bright (overexposed) or too dark (underexposed), thereby obscuring the image data. Once the fixation points are determined, the exposure around the fixation points can be adaptively corrected using image processing techniques. The resulting image is a type of foveal image in which the area of ​​the image aligned with the fovea, i.e., the center of the user's retina, is rendered in more detail than other areas in the user's field of view, for example, by changing the exposure settings of the pixels in the region of interest. As a result, image enhancement with adaptive exposure using the techniques disclosed herein can enhance human vision to facilitate night vision, low-light vision, and other visual experiences.

[0014] Figure 1A shows a first example of a digital HDR image 10 projected onto an LDR display according to a possible embodiment of the present disclosure. The digital HDR image 10 includes a person's image which becomes a fixation point 12 when viewed by an observer of the digital HDR image 10. The digital HDR image 10 is backlit by a light source 14, for example, the sun, causing the person's head in the HDR image 10 to be obscured by shadow, i.e., underexposed. On the other hand, the brightness of the sun creates an overexposed area in the digital HDR image 10. The eye-tracking area 16 includes a fixation point 12 indicated by an eye icon. The target area 18 is properly exposed.

[0015] Figure 1B shows a magnified view of the target region 18 of a digital HDR image 10 according to a possible embodiment of the present disclosure. In Figure 1B, the fixation point 12 is projected onto the target region 18, and the target region 18 is properly exposed. The conversion of the 360 ​​image from equirectangular projection to gynographic projection may be performed by a graphics processing unit (GPU). The exposure of the eye-tracking region 16 may then be modified to adapt to the local exposure of the target region 18.

[0016] Figure 1C shows a Gaussian mask 19 applied to a fixation point 12 projected onto a target region 18, according to a possible embodiment of the present disclosure. The Gaussian mask 19 may be used to smooth the image and fuse the fixation point 12 into the target region 18.

[0017] Figure 2A shows a second example of a digital HDR image 20 projected onto an LDR display according to a possible embodiment of the present disclosure. The HDR image 20 is a 14-bit raw image of a night sky and lakeside landscape displayed on an 8-bit LDR display with default exposure. The night sky includes a bright area 21. Due to the bright area 21, the default exposure renders the area within the dashed box 22 as an exemplary area in the image 20, i.e., a black area lacking detail, or an underexposed area. The underexposure results in a dark area below the night sky, where the details are not visible to the user.

[0018] FIG. 2B shows a fixation point 24 that coincides with the dashed box 22 according to a possible embodiment of the present disclosure. The fixation point 24 is identified by a gaze tracking device that follows the line of sight regarding the fixation position of the observer's eye. The fixation point 24 and the HDR image 20 can be supplied to the GPU. The GPU can then be configured to generate a Gaussian mask 26 around the fixation point 24. Using the Gaussian mask, the GPU applies a localized adaptive exposure to correct the pixels of the HDR image 20 included within the dashed box 22. This is further explained with respect to operations 1008-1012 (FIG. 10) of method 1000.

[0019] FIG. 2C shows an enlarged image portion 29 with adaptive exposure as a result of an adaptive exposure process, for example, described by an embodiment of the present disclosure. The image portion 29 is formed by a set of corrected pixels, which shows the reflection of trees on the water surface under the lakeside in the HDR image 20.

[0020] FIG. 2D shows a foveated HDR image 30 with adaptive exposure according to a possible embodiment of the present disclosure. The foveated HDR image 30 with adaptive exposure can be formed by replacing the image portion 29 with the area within the dashed box 22. The foveated HDR image 30 with adaptive exposure can then be rendered on an LDR display using appropriate details of the previously underexposed areas. Using adaptive exposure, the observer can see through the darkness to reveal details of the trees and the reflection.

[0021] FIGS. 3, 4, 5, 6A, 6B, 7A, and 7B show embodiments of an LDR display in the form of VR / AR glasses. The VR / AR glasses include an external-facing camera capable of creating HDR images such as HDR images 10 and 20. The VR / AR glasses also include a gaze tracking device for determining the fixation point 24.

[0022] Figure 3 shows a user with a wearable example of an LDR display, for example, a head-mounted wearable display 100 in the form of smart glasses or VR / AR glasses. In some embodiments, the head-mounted wearable display 100 may be in the form of VR / AR goggles or other alternative styles of headsets. The head-mounted wearable display 100 includes display capabilities, eye / gaze tracking capabilities and computing / processing capabilities. Figure 4 is a front view of the exemplary head-mounted wearable display 100 shown in Figure 3, and Figure 5 is a rear view. The exemplary head-mounted wearable display 100 includes a frame 110. The frame 110 includes a front frame portion 120 and a pair of temple arm portions 130 rotatably coupled to the front frame portion 120 by their respective hinge portions 140. The front frame portion 120 includes a rim portion 123 surrounding each optical portion in the form of a lens 127, and a bridge portion 129 connecting the rim portion 123. The temple arm portion 130 is coupled to the front frame portion 120 at the periphery of each rim portion 123, for example, in a pivotable or rotatable manner. In some examples, the lens 127 is a corrective / prescription lens. In some examples, the lens 127 is an optical material including glass and / or plastic portions that do not necessarily incorporate corrective / prescription parameters.

[0023] In some examples, the head-mounted wearable display 100 includes a display device 104 that can output visual content, for example, at an output coupler 105, so that the visual content can be visible to the user. In the examples shown in Figures 4 and 5, the display device 104 is located on one of the two arm sections 130 for illustrative purposes only. The display device 104 may be located on each of the two arm sections 130 to provide binocular output of content. In some examples, the display device 104 may be a see-through near-eye display. In some examples, the display device 104 may be configured to project light from a display light source onto a portion of teleprompter glass that acts as a beam splitter mounted at a certain angle (e.g., 30–45 degrees). The beam splitter may have reflectance and transmittance values ​​that allow light from the display light source to be partially reflected while the rest of the light is transmitted. Such an optical design may allow the user to see both the content output by the display device 104 (e.g., digital images, user interface elements, virtual content, etc.) and, for example, physical items of the world through the lens 127. In some embodiments, a waveguide optical system may be used to project content onto the display device 104. In some embodiments, the display device 104 may include an organic light-emitting diode (OLED) display configured to reproduce an image.

[0024] In some examples, the head-mounted wearable device 100 includes one or more of the following: an audio output device 106 (e.g., one or more speakers), a lighting device 108, a sensing system 111, a control system 112, at least one processor 114, an eye-tracking device 115, and a head-mounted outward-facing image sensor, such as a camera 116. In some embodiments, the camera 116 is referred to as an outward-facing camera, or egocentric camera, in contrast to an inward-facing image sensor / camera such as the eye-tracking device 115. One or more of the eye-tracking device 115 and the camera 116 may be powered by a battery housed within the frame of the head-mounted wearable display 100. The battery may be, for example, a lithium-ion rechargeable battery. In some examples, the sensing system 111 may include various sensing devices, and the control system 112 may include various control system devices, such as one or more graphics processing units (GPUs) 114 operably coupled to components of the control system 112. In some examples, the control system 112 may include a communication module, such as an RF headset transceiver, that provides communication and exchange of information between the head-mounted wearable device 100 and other external devices. In some embodiments, the transceiver includes a receiver and transmitter configured to operate in different bands or frequency ranges depending on the type or location of the external device. For example, the headset may communicate with a hand gesture sensing device 202 using short-range signals, such as Bluetooth®, or with a server computing system 1150 using long-range RF signals such as WiFi or 4G / 5G.

[0025] The eye-tracking device 115 is configured to detect and track the direction and movement of the eye's gaze. The data captured by the eye-tracking device 115 may be processed to detect and track the direction and movement of the gaze as user input. In the examples shown in Figures 4 and 5, the eye-tracking device 115 is located on one of the two arm sections 130 for illustrative purposes only. In the exemplary arrangement shown in Figures 4 and 5, the eye-tracking device 115 is located on the same arm section 130 as the display device 104, so that the user's gaze can be tracked not only with respect to objects in the physical environment but also with respect to content output for display by the display device 104. In some examples, the eye-tracking device 115 may be located on each of the two arm sections 130 to provide gaze tracking for each of the user's two eyes. In some examples, the display device 104 may be located on each of the two arm sections 130 to provide binocular display of visual content.

[0026] Figures 6A, 6B, 7A, and 7B illustrate the operation of an exemplary eye-tracking device 115. Figures 6A and 6B are partial perspective views of an exemplary eye-tracking device 115, provided on one of the two temple arm portions 130 of a head-mounted wearable display 100, simply to facilitate explanation and illustration. As described above, the eye-tracking device 115 may be provided on each of the two temple arm portions 130. Figures 7A and 7B are schematic diagrams of the operation of the eye-tracking device 115, corresponding to Figures 6A and 6B, respectively.

[0027] In this example, the eye-tracking device 115 includes an image sensor 117 (e.g., a camera) and a light source 119. In some examples, the lens 127 may include a reflective portion. The image sensor 117 may capture an image of the user's eye based on the reflection of the user's eye on the reflective portion of the lens 127. In some examples, the reflective portion of the lens 127 may be defined by a reflective coating applied to the lens 127. In some examples, the reflective coating may be made from a material that provides reflective properties but does not obstruct the user's field of view through the lens 127. For example, the reflective coating may be a near-infrared coating material. In some examples, capturing the reflective image of the user's eye may be facilitated by illuminating the user's eye. As shown in Figures 6A and 7A, the light source 119 may emit light toward the lens 127 of the head-mounted wearable display 100. The light emitted by the light source 119 may be reflected toward the user's eye by the lens 127, for example by the reflective portion of the lens 127, to illuminate the user's eye. As shown in Figures 6B and 7B, the image sensor 117 can capture an image of the user's illuminated eye reflected by, for example, the reflective portion of the lens 127. The light source 119 may emit light invisible to the user so as not to distract or cause discomfort while the head-mounted wearable display 100 is being worn. For example, the light source 119 may emit infrared light so as not to be visible to the user.

[0028] Figure 8 is a block diagram of a system 800 that creates a foveal image with adaptive exposure according to a possible embodiment of the present disclosure. The elements of system 800 include data generation components, such as those shown in Figures 3, 4, 5, 6A, 6B, 7A, and 7B, which are components of a VR / AR system. In some embodiments, system 800 includes an outward-facing camera 116, an eye-tracking device 115, an inertial measurement unit (IMU) 150, a graphics processing unit (GPU) 114, and a display 100. The outward-facing camera 116, the eye-tracking device 115, and the IMU 150 are communicatively coupled to the GPU 114 to provide input data to the GPU 114 for processing. The IMU 150 provides information regarding the position of a head-mounted display (HMD). The GPU 114 is coupled to the display 100 to provide processed image data for rendering to the display 100.

[0029] In some embodiments, the components of system 800 are not part of the VR / AR system. Instead, system 800 may include separate components, and the GPU may be provided as a processor 1132 within computing system 1100, as described below. In some embodiments, one or more components of system 800 may be located remotely from one another.

[0030] Figure 9 shows a data flow 900 of a system 800 that generates a foveal image with adaptive exposure according to a possible embodiment of the present disclosure. The data flow 900 relates to data elements shown in Figures 1A, 1B, 1C, 2A, 2B, 2C, and 2D, which are shared among the components of the system 800. The data elements include a digital image, e.g., a digital HDR image 20 showing overexposed and / or underexposed areas; a fixation point 24; an image portion, e.g., an image portion 29 containing corrected pixels; an HMD position 902 provided by the IMU; and a foveal image, e.g., a foveal image 30 with adaptive exposure. The digital HDR images 10 and 20 and the fixation point(s) 24 are sent to the GPU 114 as data input. The GPU 114 generates the foveal image 30 with adaptive exposure by performing various image processing operations, as described below with reference to Method 1000. In other words, the GPU 114 generates a foveal image 30 with adaptive exposure by increasing the exposure in some areas of the digital image 20 to brighten shadows and decreasing the exposure in other areas of the digital image 20 to darken brightness. The foveal image 30 with adaptive exposure can then be rendered for display on a display device 104 associated with a head-mounted wearable display 100.

[0031] Figure 10 shows a method 1000 for creating a foveal image with adaptive exposure, according to possible embodiments of the present disclosure. The operations of method 1000 may be performed in a different order, or not, depending on the particular application. Method 1000 may be performed using the apparatus shown in Figures 3, 4, 5, 6A, 6B, 7A, 7B, 8, and 9. Method 1000 includes preparatory actions that may occur during or after a VR / AR experience. Note that method 1000 may provide spatially adaptive exposure to some, but not all, of the input image. Therefore, it should be understood that additional processes may be provided before, during, or after method 1000, and some of these additional processes may be briefly described herein.

[0032] Method 1000, in 1002, includes identifying a digital image that is poorly exposed, according to possible embodiments of the present disclosure. Image identification may be automated by evaluating pixel intensity values ​​and determining whether, when displayed on an LTR display, the image contains blocks of extreme pixel intensity values, such as black or white areas that do not encompass the entire range of grayscale tones. Alternatively, image identification may include assessing the user's gaze to infer areas of interest, as described below. Other methods for identifying a suitable image include detecting faces that may be in shadow, detecting illegible text or signs, or detecting other content that may be recognized by a poorly exposed sensor. In some embodiments, the image being evaluated may be a still digital image or a frame of digital video. In some embodiments, the image being evaluated may be a real-time streaming video image, a 3D composite scene including meshes, avatars or virtual objects, or a 3D animation that is part of a VR / AR experience.

[0033] Method 1000, in 1004, further includes tracking the user's eye movements to determine a fixation point, according to possible embodiments of the present disclosure. Tracking eye movements is achieved using an eye-tracking device 115, as described above with respect to Figures 3, 4, 5, 6A, 6B, 7A, and 7B. A fixation point 24 is identified when the eye-tracking device 115 detects a substantially stationary gaze vector, i.e., when the user's eye movements are observed to be fixed on a particular region of interest in the input image, for example, a dashed box 22 in a digital HDR input image 20. A substantially stationary gaze can be determined with respect to statistical eye movements.

[0034] Method 1000 further includes configuring the GPU in 1006 to apply a Gaussian mask 26 around a fixation point 24 according to a possible embodiment of the present disclosure. The Gaussian mask 26 helps determine the amount of image to be processed around the fixation point. A Gaussian distribution has a central peak and decreases exponentially in all directions away from the peak, with a width depending on the standard deviation of the points in the distribution. In this context, the Gaussian mask 26 superimposes a Gaussian distribution onto the pixels in the neighborhood of the fixation point 24, thereby selecting pixels within the radius of the fixation point according to the statistics of the pixel intensity values. In some embodiments, the Gaussian mask covers less than about 10% of the area of ​​the digital HDR input image 20.

[0035] Method 1000 further includes, in 1008, calculating the average pixel intensity for each pixel in the Gaussian mask 26 according to possible embodiments of the present disclosure. The pixel intensity may be calculated by weighting the red, green, and blue (RGB) components of the pixel color according to the standard formula, Pixel Intensity = 0.299 * Red + 0.587 * Green + 0.114 * Blue. The average pixel intensity may be calculated by averaging the square blocks of pixels, e.g., 1x1, 2x2, 4x4, 8x8, etc., using a cumulative sum.

[0036] Method 1000 further includes, in 1010, correcting the exposure of pixels in a Gaussian mask 26 based on calculated pixel intensity, according to possible embodiments of the present disclosure. If the calculated pixel intensity is excessively bright, i.e., exceeds a highlight threshold level (e.g., an upper threshold level), the pixel may be corrected by reducing the exposure to below the highlight threshold. The highlight threshold level may be the highest value in the intensity range or within about 10% of the highest intensity. If the calculated pixel intensity is excessively dark, i.e., below a shadow threshold (e.g., a lower threshold level), the pixel may be corrected by increasing the exposure to a level above the shadow threshold. The shadow threshold level may be the lowest value in the intensity range or within about 10% of the lowest intensity. In some embodiments, the relative exposure may be determined as the average pixel intensity / reference intensity, e.g., the maximum pixel intensity such as 256. The extent to which the pixel exposure is increased or decreased may be determined according to the pixel position relative to the Gaussian mask 26. For example, pixels located near the outer edge of the Gaussian mask 26 may require less adjustment than pixels located near the center of the Gaussian mask 26.

[0037] In some embodiments, instead of correcting exposure, the GPU may achieve a similar effect by adjusting contrast values, for example, when displaying text.

[0038] Method 1000 further includes replacing corrected pixels from operation 1010 in the input image 20 in order to form a foveal image 30 with adaptive exposure according to a possible embodiment of the present disclosure.

[0039] Method 1000 can be employed in systems other than head-mounted VR / AR systems. For example, system 800 may include video systems such as television-based systems, teleconferencing systems, computer-based video systems, car-based video systems, e-book reading devices, GPS-based mapping programs that rely on immersive Street View images, mobile robot vision systems, camera arrays used for precision motion capture, and interactive touch displays incorporating cameras.

[0040] Figure 11 shows a computer system 1100, including a server computing system 1150 that performs functions involving a network, such as the Internet, according to some embodiments of the present disclosure. System 1100 includes a computing system 1102. The computing system 1102 may be called a client computing device or client device. The computing system 1102 is a device having an operating system 1110. In some examples, the computing system 1102 includes a personal computer, a mobile phone, a tablet, a netbook, a laptop, a smart home appliance (e.g., a smart TV), or a wearable computing device. The computing system 1102 may be any computing device having input devices 1130 such as a mouse, a trackpad, a touchscreen, a keyboard, a virtual keyboard, or a camera. The computing system 1102 may include output devices 1124 such as a display (monitor, touchscreen, etc.) that allows a user to view and select displayed content. The computing system 1102 may include one or more processors, such as a CPU / GPU 1132, which are formed on a substrate and configured to execute one or more machine-executable instructions, or parts of software, firmware, or a combination thereof. The processors, such as a CPU / GPU 1132, may be semiconductor-based; that is, the processor may include semiconductor materials capable of executing digital logic. The computing system 1102 may also include one or more memory devices 1104. The memory device 1104 may include main memory that stores information in a format that can be read and / or executed by the CPU / GPU 1132. The memory device 1104 may store applications or modules (e.g., an operating system 1110, an application 1112, an adaptive exposure application 1116, a browser application 1118, etc.) that perform specific operations when executed by the CPU / GPU 1132.For example, the adaptive exposure application 1116 may be executed by the CPU / GPU 1132 to perform the operation of method 1000.

[0041] The operating system 1110 is system software that manages computer hardware and software resources and provides common services to computing programs. In some examples, the operating system 1110 can operate to run on personal computers such as laptops, netbooks, or desktop computers. In some examples, the operating system 1110 can operate to run on mobile computers such as smartphones or tablets. The operating system 1110 may include several modules configured to provide common services and manage the resources of the computing system 1102. The computing system 1102 may include one or more input devices 1130 that enable a user to select content. Non-exclusive exemplary input devices 1130 include keyboards, mice, touch-sensitive displays, trackpads, trackballs, etc. The computing system 1102 may include one or more output devices 1124 that enable a user to view web pages and / or accept audio or other visual output.

[0042] The computing system 1102 may include applications 1112 that represent specially programmed software configured to perform various functions. One of these applications may be a browser application 1118. The browser application 1118 may be configured to display web pages, run web applications, and so on. The browser application 1118 may include additional functions in the form of extensions. In some embodiments, the browser application 1118 may also be the operating system 1110 of the computing system 1102, such as Chrome OS. The browser application 1118 may include local save location storage 1126. The local save location storage 1126 may be a data store where save locations (bookmarks, favorites, internet shortcuts, etc.) are stored.

[0043] In some embodiments, the local storage location 1126 may be associated with a user profile. In other words, more than one user may access the computing system 1102 and use the browser application 1118. In such scenarios, the local storage location 1126 may be associated with a user profile so that each user of the browser application 1118 may have their own separate local storage location 1126. In some embodiments, the user may choose to synchronize the storage location. Synchronization of the storage location may be initiated by the user on the computing system 1102. After synchronization of the storage location is initiated on the computing system 1102, the local storage location 1126 may be shared with the user's user account 1160 on the server computing system 1150.

[0044] In some examples, computing system 1102 may communicate with server computing system 1150 via network 1140. Server computing system 1150 may be one or more computing devices taking the form of several different devices, such as a standard server, a group of such servers, or a rack server system. In some examples, server computing system 1150 may be a single system sharing components such as a processor and memory. Network 1140 may include the Internet and / or other types of data networks (e.g., a local area network (LAN), a wide area network (WAN), a cellular network, a satellite network, or other types of data networks). Network 1140 may also include any number of computing devices configured to receive and / or transmit data within network 1140 (e.g., computers, servers, routers, network switches, etc.). Network 1140 may further include any number of wired and / or wireless connections.

[0045] The server computing system 1150 may include one or more processors 1152 formed on a board, an operating system (not shown), and one or more memory devices 1154. The memory devices 1154 may represent any (or more) types of memory (e.g., RAM, flash, cache, disk, tape, etc.). In some examples (not shown), the memory devices 1154 may include external storage, for example, memory that is physically far away from the server computing system 1150 but accessible from it. The server computing system 1150 may include one or more modules or engines representing specially programmed software. For example, the server computing system 1150 may include a system for managing and accessing user accounts 1160. User accounts 1160 may include data that the user has requested to be synchronized between devices such as computing system 1102. Synchronized data may include session data 1162. Session data 1162 may allow the user to resume browsing activity after switching devices. User account 1160 may also include profile data 1164. Profile data 1164 may include information describing the user with the user's consent. Profile data 1164 may also include data that identifies the user (e.g., username and password). User account 1160 may also include synchronized location storage 1166. Location storage 1166 may be a data store for the user's saved locations across the device. For example, as part of a synchronization activity, local location storage 1126 may be sent from computing system 1102 to server computing system 1150 and stored in location storage 1166.

[0046] The technical terms used herein are for the sole purpose of describing specific embodiments and are not intended to limit the disclosed embodiments. Where used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context otherwise explicitly indicates. It should be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” where used herein, specify the presence of the described features, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, actions, elements, components, and / or groups thereof.

[0047] When an element is referred to as being “combined,” “connected,” or “responding” to another element, or as being “on top of” another element, it should be understood that there may also be elements that are directly combined, connected, or responding to, or that are on top of, or interposed to, another element. In contrast, when an element is referred to as being “directly combined,” “directly connected,” or “directly responding” to another element, or as being “directly on top of” another element, there are no interposed elements. As used herein, the term “and / or” includes any combination of one or more of the items described relating to it.

[0048] Spatial relative terms such as “beneath,” “below,” “lower,” “above,” and “upper” may be used herein to facilitate explanation in describing one element or feature in relation to other elements or features, as shown in the figures. It should be understood that spatial relative terms are intended to encompass different orientations of a device in use or operation, in addition to the orientation shown in the figures. For example, if the device in the figure is inverted, an element described as “beneath” or “downward” of other elements or features becomes “above” of those elements or features. Therefore, the term “beneath” may encompass both upward and downward orientations. The device may also be oriented in other ways (rotated 70 degrees or to other orientations), and the spatial relative descriptors used herein may be interpreted accordingly.

[0049] Exemplary embodiments of the concept are described herein with reference to schematic cross-sectional views of idealized embodiments (and intermediate structures) of the exemplary embodiments. Therefore, variations from the shapes shown in the explanatory figures are expected, for example, as a result of manufacturing techniques and / or tolerances. Thus, the exemplary embodiments of the concept described should not be construed as being limited to specific shapes of the regions shown herein, but should include, for example, shape deviations resulting from manufacturing. Accordingly, the regions shown in the figures are essentially schematic, and their shapes are not intended to describe the actual shapes of the regions of the device, nor are they intended to limit the scope of the exemplary embodiments.

[0050] Furthermore, while terms such as “first,” “second,” and so on may be used herein to describe various elements, it should be understood that these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. Thus, the “first” element may also be referred to as the “second” element without departing from the teachings of the disclosed embodiments.

[0051] Unless otherwise defined, terms used herein (including technical and scientific terms) have the same meanings as those generally understood by those skilled in the art to which the concepts belong. Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant art and / or herein, and should not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0052] While specific features of the embodiments described herein have been illustrated, numerous modifications, substitutions, alterations, and equivalents will be conceivable to those skilled in the art. Therefore, it should be understood that the appended claims are intended to encompass such modifications and alterations that fall within the scope of the embodiments. They are presented only as examples, not as limitations, and various modifications in form and detail are permitted. Any part of the apparatus and / or method described herein may be combined in any combination, except for mutually exclusive combinations. The embodiments described herein may include various combinations and / or partial combinations of the functions, components, and / or features of the different embodiments described.

Claims

1. It is a method, Identifying digital images that have underexposed or overexposed areas, To determine the fixation point, the observer's eye movements in the digital image are tracked, Applying a Gaussian mask to the digital image surrounding the fixation point, For each pixel in the Gaussian mask, Calculating pixel intensity, When the pixel intensity is darker than the dark area threshold, the exposure setting of the pixel in the digital image is increased. When the pixel intensity is brighter than the highlight threshold, the exposure setting of the pixel in the digital image is reduced. To display the aforementioned digital image to the user, Methods that include...

2. The method according to claim 1, wherein the dark area threshold is within approximately 10% of the maximum intensity.

3. The method according to any one of claims 1 to 2, wherein the bright area threshold is within approximately 10% of the minimum intensity.

4. The method according to any one of claims 1 to 3, wherein the application of the Gaussian mask covers less than 10% of the digital image.

5. The digital image is derived from a composite 3D scene, according to the method according to any one of claims 1 to 4.

6. It is a system, The display and The system further comprises an outward-facing camera mounted on the display, wherein the outward-facing camera is configured to produce a digital image, and the system further comprises The system further comprises an eye-tracking device attached to the display, the eye-tracking device being configured to identify selected pixels of the digital image, and the system further A system comprising an outward-facing camera, an eye-tracking device, and a graphics processing unit (GPU) communicably coupled to the display, wherein the GPU is configured to adaptively correct the exposure of the selected pixels in order to produce a foveal image for projection onto the display.

7. The system according to claim 6, wherein the display is a low dynamic range (LDR) display.

8. The system according to any one of claims 6 to 7, wherein the display is a head-mounted display suitable for use with augmented reality and virtual reality.

9. The system according to any one of claims 6 to 8, wherein the digital image is a high dynamic range (HDR) image.

10. The system according to any one of claims 6 to 9, wherein the digital image has a resolution of at least 10 bits, and the display has a resolution of 8 bits.

11. The system according to any one of claims 6 to 10, wherein the eye-tracking device is an infrared eye-tracking device configured to track the user's gaze at a frequency of 120 Hz.

12. The system according to any one of claims 6 to 11, wherein the foveal image is an image of the user's field of view.

13. The system according to any one of claims 6 to 12, wherein the digital image is a frame of a digital video.

14. It is a headset, Wearable displays and An outward-facing camera attached to the wearable display, An eye-tracking device attached to the wearable display, The device comprises the outward-facing camera, the eye-tracking device, and a graphics processing unit (GPU) coupled to the wearable display, wherein the GPU is Receive image data and region of interest, To create a foveal image, the exposure of the portion of the image data corresponding to the region of interest is adjusted. A headset configured to project the aforementioned foveal image onto the wearable display.

15. The headset according to claim 14, wherein the outward-facing camera is configured to capture a 360-degree image and transmit the image data of the 360-degree image to the GPU.

16. The headset according to claim 14, wherein the outward-facing camera is configured to capture an image of a scene and transmit the image data related to the scene to the GPU.

17. The headset according to claim 16, wherein the eye-tracking device is configured to identify the region of interest in the scene, the region of interest indicating an underexposure in the image of the scene, and the eye-tracking device is configured to transmit the region of interest to the GPU.

18. The headset according to claim 17, wherein the region of interest is identified by a substantially stationary gaze vector detected by the gaze tracking device.

19. The headset according to any one of claims 17 to 18, wherein the GPU is configured to adjust the contrast values ​​of a portion of the wearable display identified by the eye-tracking device.

20. The wearable display includes augmented reality glasses or virtual reality glasses, as described in any one of claims 14 to 19.

21. The headset according to any one of claims 14 to 20, further comprising an inertial measurement unit (IMU) configured to measure the position of the wearable display and transmit the position to the GPU.

Images