Image enhancement device with gaze tracking

Abstract

The invention relates to an image enhancement device with gaze tracking. An electronic device may have a display and a camera. Control circuitry in the device may collect information about a user's gaze point using a gaze tracking system and other sensors, may collect information about real-world images by analyzing the real-world images, such as information about content, motion, and other image attributes, and may also collect information about the user's gaze point by analyzing the real-world images. User visual information such as user visual acuity, contrast sensitivity, field of view, and geometric distortion may be acquired, user inputs such as user preferences and user mode selection commands may be acquired, and other inputs may be acquired. Based on the gaze point information and / or other acquired information, the control circuit may display real-world images and supplemental information on the display. The supplemental information may include enhancements such as icons, text tags, and other computer-generated text and graphics overlaid on the real-world image, and may include augmented image content such as an amplified portion of the real-world image.

Description

[0001] Divisional Application Statement

[0002] This application is a divisional application of a Chinese patent application with an application date of February 21, 2019, an invention title of "Image Enhancement Device with Gaze Tracking", and an application number of 201980015938.1.

[0003] Cross-reference

[0004] This patent application claims the priority of U.S. Patent Application No. 16 / 280,786 filed on February 20, 2019 and Provisional Patent Application No. 62 / 643,023 filed on March 14, 2018, and these patent applications are hereby incorporated by reference in their entireties. Technical Field

[0005] The present disclosure generally relates to electronic devices, and more particularly to electronic devices for displaying mixed reality content. Background Art

[0006] The present disclosure generally relates to electronic devices, and more particularly to electronic devices for displaying mixed reality content.

[0007] A mixed reality device may have a camera and a display. The camera may capture real-world images. Then, content such as text and graphics may be overlaid on the real-world images to generate mixed reality images.

[0008] Presenting mixed reality content to a user may be challenging. Presenting a mixed reality image to a user may be disrupted, for example, by head and eye movements. Users may have different visual capabilities. If not careful, presenting a mixed reality image that causes disorientation or motion sickness and makes it difficult for the user to identify items of interest. Summary of the Invention

[0009] An electronic device may have a display and a camera. The camera may be used to capture real-world images. The real-world images may be displayed on the display for the user.

[0010] A control circuit in the device may use a gaze tracking system and other sensors to collect information about the user's gaze point. The control circuit may also collect information about the real-world images by analyzing the real-world images, such as information about content, motion, and other image attributes. User visual information such as user visual acuity, contrast sensitivity, field of view, and geometric distortion may be provided to the control circuit. An input-output circuit may be used by the control circuit to collect user inputs, such as user preferences and user mode selection commands, and may be used to collect position information and other inputs.

[0011] Based on the fixation point information and / or other acquired information, the control circuit can display supplementary information on the real-world image on the display. The supplementary information can include enhancements overlaid on the real-world image such as icons, text labels, and other computer-generated text and graphics, and can include enhanced image content such as magnified portions of the real-world image. If desired, the real-world image can be globally or locally enhanced by adjusting contrast, changing image color, enhancing edges, adjusting geometric visual distortion, and performing other image enhancements on the real-world image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram of an exemplary electronic device according to one embodiment.

[0013] Figure 2 is an illustration of a fixation tracking circuit according to one embodiment.

[0014] Figure 3 is an illustration of an exemplary real-world image and supplementary information displayed on the real-world image according to one embodiment.

[0015] Figure 4 、 Figure 5 and Figure 6 are illustrations showing exemplary image enhancement regions on a display according to one embodiment.

[0016] Figure 7 is a flowchart of an exemplary operation related to providing mixed reality content to a user according to one embodiment. DETAILED DESCRIPTION

[0017] The electronic device can be provided with a display. The display can be used to display supplementary content overlaid on real-world content. The supplementary content can include, for example, enhancements such as computer-generated text and graphics, and can include enhanced image content such as real-world image data that has been modified by adjusting image attributes such as contrast, magnification, and other characteristics.

[0018] An electronic device may use information such as information from a gaze tracking system and other information when enhancing the display of an image. For example, a portion of a real-world image may be magnified for a low-vision user only when the user's fixation point is relatively stationary to avoid causing a motion sickness effect. The electronic device may analyze the real-world image to identify content of interest (e.g., the location and movement of objects, the identification of faces and other items, the presence of text, the location of the edges of objects, etc.). If needed, information such as the user's glasses prescription and other user visual data may be used for display image enhancement. For example, if the user has impaired vision (e.g., if the user has low spatial visual acuity), the text in the image may be magnified, or the image content may be distorted to compensate for the geometric distortion caused by the user's vision. The image enhancement may be applied globally to the real-world image or may be displayed in an appropriate image enhancement region. If needed, user input and other information may also be used to display supplementary information on the real-world image.

[0019] Figure 1 A schematic diagram of an exemplary electronic device of a type that may be used to present mixed reality content to a user is shown. The electronic device 10 may be a head-mounted device (head-mounted display), a cellular phone, a tablet computer, a head-up display (e.g., a mixed reality display in an automobile or other vehicle), a laptop computer or a desktop computer, a television, a watch, or other electronic equipment. As Figure 1 shown, the electronic device 10 may have a control circuit 20. The control circuit 20 may include storage and processing circuitry for controlling the operation of the device 10. The circuit 12 may include storage devices such as hard disk drive storage devices, non-volatile memory (e.g., electrically programmable read-only memory configured to form a solid-state drive), volatile memory (e.g., static or dynamic random access memory), and the like. The processing circuitry in the control circuit 12 may be based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors, power management units, audio chips, graphics processing units, application specific integrated circuits, and other integrated circuits. Software code may be stored on the storage device in the circuit 20 and run on the processing circuitry in the circuit 20 to implement the control operations of the device 10 (e.g., operations associated with capturing a real-world image using a camera, data acquisition operations, operations involving monitoring the user's fixation point, operations involving generating content overlaid on the real-world image, operations associated with performing pattern recognition and other image processing operations on the real-world image, operations associated with displaying an image having real-world content and overlaid content, etc.).

[0020] Device 10 may include input-output circuitry 22. The input-output circuitry 22 may be used to allow the device 10 to receive data from external equipment (e.g., a computer or other electrical equipment), and to allow a user to provide user input to the device 10. The input-output circuitry 22 may also be used to collect information about the environment in which the device 10 is operating. The output component in the circuitry 22 may allow the device 10 to provide output to the user, and may be used to communicate with external electrical equipment.

[0021] As Figure 1 shown, the input-output circuitry 22 may include a camera such as camera 40. The camera 40, which may sometimes be referred to as a front camera, a scene camera, or a real-world image camera, may be used to capture images of the real world (e.g., the user's environment). The real-world images captured by the camera 40 may include the content within the user's field of view and the content located outside the user's field of view (e.g., the portion of the user's environment adjacent to the user's field of view).

[0022] The input-output circuitry 22 may also include a display, such as display 14. The display 14 may be used to display images for a user of the device 10. The display 14 may be an organic light-emitting diode display, a liquid crystal display, a liquid crystal on silicon display, a micro-mirror array display (e.g., a micro-electro-mechanical systems (MEMS) display, sometimes referred to as a digital micro-mirror device), or any other suitable display. The display 14 may be mounted in the device 10 such that the image on the display 14 is provided within the user's field of view. In some configurations, the display 14 may be supported on a head-mounted support structure (glasses, helmet, hat, goggles, etc.). In other configurations, the display 14 may be mounted in a vehicle or other equipment (e.g., in a position where the image output from the display can be reflected from the inner surface of a window into the user's field of view to present the image to the user).

[0023] The display 14 may be directly in the user's line of sight, or an optical combiner in the user's line of sight may be used to merge the image from the display 14 with the user's view of the surrounding environment of the device 10. In some arrangements, the display 14 is housed within the housing of a cellular phone, a tablet computer, or other portable electronic device. Generally, the display 14 may be placed in any suitable position for providing an image to the user. During operation, the user may view real-world image content presented through the display 14 (e.g., real-world images captured by the camera and displayed on the display 14) and / or may directly view real-world content (e.g., when the real world is visible through the optical combiner that is being used to present the image from the display 14 to the user). Exemplary configurations in which the user views real-world content overlaid with supplementary content on the display 14 are sometimes described herein as examples.

[0024] The input-output circuit 22 may include a gaze tracking system, such as the gaze tracking system 16. The gaze tracking (eye monitoring) system 16 may include an image sensor, a light source, and / or other equipment for monitoring the user's eyes. The gaze tracking system 16 may include, for example, one or more visible and / or infrared cameras that face the user's eyes and capture images of the user's eyes. During operation of the device 10, the control circuit 20 may use the gaze tracking system 16 to track the user's gaze. The cameras and / or other sensors in the system 16 may, for example, determine the position of the user's eyes (e.g., the center of the user's pupil), and may determine the direction of the user's eye orientation (the direction of the user's gaze, sometimes referred to as the measured point of regard). By processing this information and information about the position of the display 14, the position of the user's point of regard may be dynamically determined. If desired, additional information (e.g., information from an inertial measurement unit that includes an accelerometer, a compass, and / or a gyroscope and / or information from other motion sensors) may be used to determine the position of the user's point of regard. For example, the inertial measurement unit, a visual ranging sensor in the circuit 22, and / or other sensors may determine the orientation of the device 10 and / or the user's head relative to the surrounding environment, and this information may be used in addition to the gaze tracking information to determine the user's current point of regard.

[0025] Sensors in the input-output device 18 and other input devices may also be used to collect user input and other information. The input-output device 18 may include, for example, position and motion sensors (e.g., a compass, a gyroscope, an accelerometer, and / or other devices for monitoring the position, orientation, and movement of the device 10), may include force sensors, temperature sensors, touch sensors, buttons, capacitive proximity sensors, light-based proximity sensors, other proximity sensors, color ambient light sensors and other ambient light sensors, strain gauges, gas sensors, pressure sensors, humidity sensors, magnetic sensors, gesture sensors, depth sensors (e.g., structured light sensors for three dimensions and other depth sensors), and other sensors, may include audio components (such as a microphone) for collecting voice commands and other audio inputs, and may include speakers for providing audio output (e.g., for providing sound to the user's left and right ears). If desired, the input-output device 18 may include tactile output devices (e.g., vibration components), light-emitting diodes, lasers, and other light sources, and other output components.

[0026] Circuit 22 may include wired and wireless communication circuits that allow device 10 (e.g., control circuit 20) to communicate with external equipment (e.g., remote controls, joysticks, and other input controllers, portable electronic devices, computers, displays, etc.) and allow signals to be conveyed between components (circuits) at different locations of device 10. A Global Positioning System (GPS) receiver circuit and / or other satellite navigation system circuits in input-output circuit 22 and / or other position sensor circuits in circuit 22 may be used to determine the user's position and / or speed. Using the wireless communication circuit in circuit 22, device 10 may collect information (e.g., weather information, traffic conditions, etc.) via a network such as the Internet

[0027] Figure 2 is an illustration showing how the gaze tracking system 16 may collect eye position information about the user's eyes 30. In a typical scenario, the gaze tracking system 16 may include components for simultaneously monitoring both eyes (such as Figure 2 the eyes 30).

[0028] As Figure 2 shown, the system 16 may include one or more image sensors, such as the gaze tracking camera 24. Each camera 24 may be focused in front of the user's eyes (such as eyes 30) so that characteristics of the user's eyes can be measured. One or more light-emitting diodes, lasers, lights, and / or other light-emitting components may be used to form a light source for the gaze tracking system 16 (see, for example, light source 26).

[0029] During operation, the light source 26 may emit light 28. For example, the light source 26 may emit a plurality of light beams 28 (e.g., 2 - 10 light beams, 6 light beams, 4 - 7 light beams, 4 - 9 light beams, etc.) towards the user's eyes 30. The light 28 may be reflected off the surface of the eyes 30. The camera 24 may collect an image of the eyes 30. The control circuit 20 may use information about the appearance of the eyes 30 (e.g., iris information, pupil information, blood vessel information, etc.) and / or information from the reflected light (e.g., one or more light beams) from the cornea 32 and other parts of the eyes 30 to determine the position of the pupil center 36 of the pupil 34 and the direction of the user's current gaze (gaze direction 38). The control circuit 20 may use the eye position information (pupil center information, eye orientation, etc.) collected by the gaze tracking system 16, the resulting information determined in the user's gaze direction 38, and information about the position of real-world objects relative to the system 16 and the eyes 30 (e.g., items in the environment around device 10) to dynamically identify the user's gaze point within the real-world environment around device 10. Then, enhancements and / or augmentations may be provided to the real-world image based on the gaze point (e.g., the position in the real world where the user is gazing).

[0030] During operation, the control circuit 20 may use the camera 40 to capture real-world images and may use the gaze tracking system 16 and other input-output devices 18 to collect user input and information about the user's environment. Data about the user, such as the user's glasses prescription and other information about the user (e.g., visual impairment details such as low vision details), may also be collected. This information may be used by the control circuit 20 to display mixed reality content for the user on the display 14. For example, text in the real-world image may be magnified to accommodate a low vision user. Contrast and other image attributes may also be adjusted.

[0031] If desired, the control circuit 20 may obtain content attributes, such as the identity of a person in the real-world image (e.g., by performing pattern recognition operations). Mixed reality content may be presented to the user based on these content attributes. For example, the name of a person in the user's field of view may be presented to the user by overlaying a text box on the real-world image. The text box containing the name of each person in the user's field of view may be placed adjacent to that person, or only those people near the user's fixation point may be labeled. Fixation point information may be used to determine which people the user is observing.

[0032] Gaze point information (e.g., information about the position and movement of the user's gaze point) may also be analyzed to determine when supplementary content should be presented based on real-world image content outside of the user's field of view (or only in the user's extreme peripheral vision). For example, if the user's gaze point is moving to the left, an icon indicating that interesting content (e.g., a moving vehicle) is located exactly to the left of the user's field of view may be presented to the user. In this way, advance notice of objects that are not currently being viewed by the user may be provided to the user.

[0033] For example, consider Figure 3 the real-world image 50. The real-world image 50 may be captured by the input-output circuit 22 using the camera 40. If desired, the field of view of the camera 40 may be larger than the user's field of view (see, e.g., the user's field of view 52). The image 50 may include real-world objects such as objects 58, 56, and 54 (e.g., people, vehicles, buildings, furniture, landscape elements, etc.). During operation of the device 10, mixed reality content may be presented to the user in some or all of the user's field of view. For example, computer-generated images (still and / or moving image content with labels and other text and / or graphics) and / or portions of the real-world image 50 that have been enhanced (e.g., by image processing to adjust magnification, contrast, color, brightness, edge details, etc.) may be presented in one or more regions, such as in the supplementary image region 60 (sometimes referred to as the supplementary content region, the overlay region, etc.).

[0034] For example, the information presented in region 60 may include a portion of the real-world image 50 that has been enhanced through image processing. The portion of the real-world image 50 selected for image enhancement may be selected based on user input and other information. For example, fixation point information may be used to determine which portion of the real-world image 50 will be magnified or otherwise enhanced for display in region 60. For example, if the user is viewing object 58 and the user's fixation point overlaps object 58 (see, e.g., fixation point 62), then a portion of the real-world image 50 associated with some or all of object 58 may be enhanced (e.g., magnified, having additional contrast, etc.), and the resulting enhanced image may be displayed in region 60. If desired, when the user's fixation point does not overlap object 58 but is within a predetermined distance of object 58, some or all of the portion of the image covering object 58 may be enhanced. For example, control circuit 20 may display enhanced image content in response to detecting that the user's fixation point is within region 66 adjacent to object 62 (see, e.g., fixation point 64).

[0035] When determining when the user's fixation point overlaps a desired object or a region near the desired object, control circuit 20 may evaluate how long the user's fixation has remained at different positions in the image. A threshold fixation dwell time or other suitable criterion may be applied to the fixation point information to determine whether the user's fixation point has remained in a particular position long enough to trigger the display of the content in region 60. For example, in response to detecting that the user's fixation point has remained within region 66 for more than at least 0.2 s, at least 0.5 s, at least 1 s, at least 2 s, at least 4 s, at least 10 s, less than 8 s, less than 5 s, or other suitable time, a magnified version of object 62 may be presented in region 60.

[0036] In some configurations, control circuit 20 monitors the movement of the user's fixation point and takes an action based on the fixation point movement. In Figure 3In the example, if the user's fixation point 68 is moving in direction 70 towards the left edge 72 of the user's field of view 52 (e.g., moving at a speed exceeding a predetermined threshold speed for a predetermined threshold amount of time), the control circuit 20 may analyze the image 50 for an object of interest located outside the field of view 52 (e.g., an item located outside the left edge 72, such as object 54). Pattern recognition operations may be performed on the image 50 to determine which elements of the image 50 correspond to the object of interest. For example, the image 50 may be processed by the control circuit 20 to identify vehicles, people, objects moving at a speed exceeding a particular speed, objects having a brightness exceeding a predetermined brightness threshold, objects of a particular color, objects containing text, etc. In response to determining that the user's fixation point 68 is moving in direction 70 and that an object of interest in the image 50, such as object 54, is located outside the user's current field of view 52 in direction 70, associated supplementary information may be presented to the user. For example, the control circuit 20 may display a graphical element, such as arrow 74, pointing to the out-of-view object, such as object 54. This directs the user's attention to potentially important objects just outside the user's field of view and helps the user to be aware of such objects. For example, if the user is driving a vehicle or walking along a street, notifying the presence of object 54 (e.g., a fast-moving vehicle) may be helpful. In some arrangements, disorientation and motion sickness may be avoided by suppressing the display of enlarged text or other supplementary content in response to detecting movement of the user's fixation point (e.g., an eye saccade).

[0037] Any suitable criteria can be used to determine when the control circuit 20 should display supplementary information on the real-world image (e.g., overlay content such as enhanced image content in region 60 and / or text or graphics such as arrow 74, text labels, warnings, etc.). For example, the control circuit 20 can determine the position of the user's fixation point and the movement of the user's fixation point by processing the fixation point position information from the gaze tracking system 16 and the information about the orientation of the user's head from position and motion sensors (orientation sensors) such as an inertial measurement unit and / or a visual ranging system. If the position and / or movement of the fixation point meet a predetermined criterion (e.g., if the fixation point stays at a specific position for more than a predetermined amount of time, if the fixation point moves at a specific speed and / or direction), then appropriate supplementary information can be displayed. As another example, supplementary information can be displayed based on information such as the position of the device 10 (e.g., the position of the device 10 determined by a satellite navigation system receiver and / or other position sensors in the input-output device 18), sensor information (e.g., information about the ambient lighting conditions indicating whether it is day or night), based on the content in the image 50 (e.g., whether there is a face and / or whether the face present corresponds to a specific individual, whether the image 50 contains a specific object, characterized by a predetermined amount of movement, having a specific brightness level or containing a specific color or pattern, etc.), and / or based on other information. The display of supplementary information can also be suppressed (e.g., temporarily) whenever certain criteria are met (e.g., excessive movement of the user's fixation point, etc.).

[0038] If desired, the user of the device 10 can press a button, speak a voice command, orient the user's gaze direction in a specific orientation (e.g., look down to invoke autofocus, look up and to the right, look so that the user's fixation point overlaps a displayed interactive icon, etc.), blink (e.g., blink three times in quick succession so that the camera 24 or other sensor can detect a predetermined blink command), or otherwise take an action that causes the control circuit 20 to display supplementary information.

[0039] The conditions for determining whether to display supplementary information and the type of supplementary information presented (e.g., magnified image content, contrast-enhanced image content, etc.) can include user data such as the user's glasses prescription and other visual function information. For example, the user's eyesight can be characterized using an ophthalmic instrument (e.g., to characterize the user's visual field, visual acuity, contrast sensitivity, and geometric distortion). This information can then be used to determine, for example, whether the image content should be enhanced by increasing the contrast or by increasing the magnification (or both). In some cases (e.g., when the user's fixation point is moving), presenting magnified content can cause motion sickness. Therefore, when more than a predetermined amount of fixation point movement is detected or other suitable supplementary information suppression criteria are met, the presentation of supplementary information (such as enhanced image information in region 60) can be suppressed.

[0040] Supplementary information (e.g., computer-generated text or graphics or other enhanced information and / or enhanced image content) may be displayed in any suitable portion of the user's field of view. For example, the supplementary information may be displayed in Figure 4 region 60 of Figure 5 region 60 of (e.g., along the edge of the displayed image, such as along the bottom edge of the displayed image), Figure 6 region 60 of (e.g., in the upper right corner of the displayed image or other corner of the displayed image), and / or other suitable portions of the image being displayed by display 14.

[0041] Figure 7 A flowchart illustrating exemplary operations involving using device 10 to provide a real-world image and supplementary information to a user on display 14 is shown.

[0042] During the operation of block 80, control circuit 20 may use input-output circuit 22 to acquire information regarding the user's gaze. The gaze tracking system 16 and other sensors in the inertial measurement unit and / or input-output device 18 may be used to monitor eye movement and optionally other inputs such as head movement / orientation and / or information regarding the orientation of device 10 and thereby measure the user's gaze point. Information regarding the user's gaze point (such as gaze point location and / or gaze point movement) may be analyzed to determine whether the user is providing an intentional input (gazing at a predetermined region of the displayed image for a predetermined time to intentionally invoke the display of supplementary information), and / or information regarding the user's gaze point (such as gaze point location and / or gaze point movement) may be analyzed to determine whether the user is viewing a particular item of interest or is moving towards a particular item of interest (by way of example). The operation of block 80 may include characterizing the user's eye movement as fixed (not moving), saccadic (moving), and blinking.

[0043] During operation of block 82, control circuit 20 may analyze the captured image, such as image 50. For example, image recognition algorithms and other processing algorithms may be used to process an image of the scene that the user is viewing (e.g., real-world image 50) to identify the content and other attributes in the image. For example, the image may be processed to identify image attributes such as brightness, color faults, contrast, etc. Modes may be detected (e.g., to identify vehicles, roads, people, faces, the identity of a person associated with a particular face, road signs with or without text, text on billboards, and other advertisements and / or other items in the user's environment). Edge detection processes may be used to detect object edges. Stationary objects and moving objects may be classified. Techniques such as these may be used to process all the information in image 50, or if desired, a subset of the information in image 50. For example, image processing such as face recognition processing may be performed on those faces where the user's gaze point stays, but the image processing may be skipped for other faces to reduce the processing load. The processing of this information may also be affected by the outputs of other sensors. For example, the road sign information is processed and recognized for magnification only when the user and device 10 are moving at a speed faster than a predetermined speed (indicating that the user is driving).

[0044] During operation of block 84, information about vision and other attributes of the user may be collected. For example, an ophthalmic instrument may be used to collect user data such as vision impairment information and provide it to control circuit 20 (e.g., via a wired and / or wireless communication path, using a user input device, etc.). By characterizing the user's visual acuity, contrast sensitivity, visual field, geometric distortion, and other visual attributes, image enhancement and strengthening of appropriate targets may be provided for the user.

[0045] During operation of block 86, user input may be collected using input-output circuit 22. For example, the gaze tracking system 16 may collect intentional eye inputs such as blinks and intentional gaze point inputs (e.g., eye gestures such as gestures involving the user's gaze point moving into and out of a particular area or moving the intentional direction of the user's gaze downward to automatically call up an enlarged image 50 that serves as a digital bifocal, etc.). Circuit 22 may also use buttons, touch sensors, microphones, and other input-output circuits to collect user input. The user input may be used to place device 10 in different operation modes (e.g., walking mode, in which supplementary information is provided at the center of the displayed image; driving mode, in which supplementary information is provided along the edge of the displayed image; meeting mode, in which face recognition operations are used to identify and label the participants in a meeting; inactive mode, in which no supplementary information is displayed, etc.). The user input may also be used to provide preferences for device 10 (e.g., the user may adjust settings such as a preferred display brightness level, threshold, magnification settings, the desired position of supplementary information area 60, etc.).

[0046] Additional inputs may be acquired by the input-output circuit 22 during operation of block 88. For example, the control circuit 20 may use the circuit 22 to acquire position information, information about the speed and direction of movement of the device 10, information about traffic conditions, information about local weather, and the like.

[0047] During operation of block 90, the control circuit 20 may display an image captured by the camera 40 on the display 14 for viewing by the user (and / or the user may view the real world through an optical combiner such as an optical combiner in a window or a head-mounted device). Supplementary information may be displayed on the display 14 to overlay the real-world content. For example, the information may be displayed on some or all of the display 14, and the information includes enhanced visual information such as graphics, text, and other computer-generated overlays and / or includes all or some of the enhanced visual information such as the image 50, which has been processed to adjust contrast, to adjust brightness, to adjust color, to adjust object features to enhance edges by applying edge enhancement, to adjust magnification, to adjust geometric distortion, or to adjust other image attributes. User data acquired during operation of block 84 (e.g., details of the user's visual impairment) may be considered during operation of block 90. For example, if the user's vision is characterized by geometric distortion, compensation for geometric distortion may be applied when displaying the image on the display 14. If the user has low visual spatial acuity, the content may be appropriately magnified.

[0048] Generally, the supplementary information may be displayed based on the information acquired during block 80 (e.g., fixation point information), the image analysis performed during block 82, the information acquired during block 84, the information acquired during block 86 (e.g., mode selection information and other information provided as user input), and / or the information acquired during operation of block 88 (e.g., information about the current weather, traffic conditions, etc.). Local adjustments may be made to the image (e.g., magnifying an appropriate area such as Figure 4 、 Figure 5 and Figure 6 the text in the area 60 of

[0049] According to one embodiment, an electronic device is provided, the electronic device including: a camera configured to capture an image of the real world; a display configured to display the image; a fixation tracking system configured to acquire fixation point information; and a control circuit configured to display magnified supplementary content in the image on the display based on the fixation point information.

[0050] According to another embodiment, the magnified supplementary content includes a magnified portion of the image of the real world.

[0051] According to another embodiment, the control circuit is configured to display the magnified portion of the image of the real world as an overlay on a region of the image of the real world.

[0052] According to another embodiment, the region is characterized by a position within the image of the real world, and the electronic device includes an input-output circuit configured to collect user input to adjust the position.

[0053] According to another embodiment, the control circuit is configured to display the magnified portion in response to a fixation gesture.

[0054] According to another embodiment, the control circuit is configured to display the magnified supplementary content in response to user blink input collected by the gaze tracking system.

[0055] According to another embodiment, the control circuit is configured to display the magnified supplementary content in response to determining from the gaze point information that the gaze point has remained on a portion of the image for longer than a predetermined gaze point dwell time.

[0056] According to another embodiment, the control circuit is configured to display an enhancement including computer-generated graphic content in the image on the display based on the gaze point information.

[0057] According to another embodiment, the control circuit is configured to display the magnified supplementary content based on information about the user's vision attributes.

[0058] According to another embodiment, the control circuit is configured to analyze the image to identify objects in motion.

[0059] According to another embodiment, the control circuit is configured to analyze the image to identify a face, and the control circuit is configured to display text that labels the identified face with an identity associated with the identified face.

[0060] According to another embodiment, the control circuit is configured to analyze the image to identify text, and the magnified supplementary content includes a magnified version of the text.

[0061] According to another embodiment, the electronic device includes an input-output circuit, the control circuit is configured to collect user input using the input-output circuit, and the control circuit is configured to display the magnified supplementary content based on an operation mode selected using the user input.

[0062] According to another embodiment, the input-output circuit includes a position sensing circuit, and the control circuit is configured to display the magnified supplementary content based on the geographical location identified using the position sensing circuit.

[0063] According to another embodiment, the control circuit is configured to provide the magnified supplementary information based on the fixation dwell time associated with the fixation point information, an attribute in the image, user vision attributes, and user input.

[0064] According to another embodiment, the electronic device includes a motion sensor, and the control circuit is configured to display the magnified supplementary content in response to determining that the speed measured by the motion sensor exceeds a predetermined limit.

[0065] According to one embodiment, there is provided an electronic device including: a camera configured to capture a real-world image; a display; a gaze tracking system configured to collect fixation point information; and a control circuit configured to display enhanced image content overlapping the image on the display based on the fixation point information.

[0066] According to another embodiment, the control circuit is configured to collect user vision information including user vision characteristics selected from the group consisting of visual acuity, contrast, field of view, and geometric distortion, and the control circuit is configured to display the enhanced image content based on the collected user vision information.

[0067] According to another embodiment, the enhanced image content includes an enhanced portion of the image, the enhanced portion of the image being selected from the group consisting of a magnified portion of the image; a portion of the image having increased contrast, a portion of the image having enhanced edges, a portion of the image having a changed color.

[0068] According to one embodiment, there is provided an electronic device including: a camera configured to capture a real-world image; a gaze tracking circuit configured to collect fixation point information; a display; and a control circuit configured to: analyze the real-world image to identify text in the image; and display a magnified version of the text covering a portion of the image on the display based on the fixation point information.

[0069] The foregoing is merely illustrative and various modifications may be made to the embodiments. The foregoing embodiments may be implemented independently or in any combination.

Claims

1. An electronic device, comprising: a camera configured to capture an image of the real world; a display, wherein the display is configured to display the image of the real world; a gaze tracking system configured to collect gaze point information; and a control circuit configured to display magnified supplementary content on the display based on the gaze point information and based on information about the user's vision attributes, wherein the magnified supplementary content includes a magnified portion of the image of the real world, and wherein the control circuit is configured to display the magnified portion of the image of the real world as an overlay on a region of the image of the real world.

2. The electronic device according to claim 1, wherein the region is characterized by a position within the image of the real world, and wherein the electronic device includes an input-output circuit configured to collect user input to adjust the position.

3. The electronic device according to claim 1, wherein the control circuit is configured to display the magnified portion in response to a gaze point gesture.

4. The electronic device according to claim 1, wherein the control circuit is configured to display the magnified supplementary content in response to a user blink input collected by the gaze tracking system.

5. The electronic device according to claim 1, wherein the control circuit is configured to display the magnified supplementary content in response to determining that the gaze point has remained at a specific position for longer than a predetermined gaze point dwell time based on the gaze point information.

6. An electronic device, comprising: a camera configured to capture an image of the real world; a gaze tracking system configured to collect gaze point information; and a display configured to display magnified supplementary content on the display based on the gaze point information, wherein the magnified supplementary content includes a magnified portion of the image of the real world, and wherein the display is configured to stop displaying the magnified supplementary content in response to detecting motion in the gaze point information.

7. The electronic device according to claim 6, wherein the motion includes an eye saccade.

8. The electronic device according to claim 6, further comprising: a motion sensor; and a control circuit configured to display the magnified supplementary content in response to determining that a speed measured by the motion sensor exceeds a predetermined limit.

9. The electronic device according to claim 8, wherein the control circuit is further configured to display the magnified supplementary content based on user vision impairment information, the user vision impairment information including information for characterizing attributes selected from the group consisting of visual acuity, contrast sensitivity, visual field, and geometric distortion.

10. The electronic device according to claim 9, wherein the magnified supplementary content is configured to compensate for the attributes.

11. An electronic device, comprising: a camera configured to capture an image of the real world; A display configured to display the image of the real world; A gaze tracking system configured to collect gaze point information; and A control circuit configured to display supplementary content on the display based on the gaze point information, wherein the supplementary content includes a modified portion of the image of the real world, and wherein the control circuit is configured to display the modified portion of the image of the real world as an overlay on the image of the real world.

12. The electronic device according to claim 11, wherein the modified portion includes a magnified portion of the image of the real world.

13. The electronic device according to claim 11, wherein the modified portion includes a portion of the image of the real world having enhanced contrast.

14. The electronic device according to claim 11, wherein the modified portion includes a brighter portion of the image of the real world.

15. The electronic device according to claim 11, further comprising a position sensing circuit, wherein the control circuit is configured to display the supplementary content based on a geographical location identified using the position sensing circuit.

16. An electronic device comprising: A camera configured to capture an image of the real world; A display configured to display magnified content; An optical combiner configured to present the magnified content from the display, wherein the real world is visible through the optical combiner; A gaze tracking system configured to collect gaze point information; and A control circuit configured to display the magnified content on the display based on the image of the real world, the gaze point information, and as an overlay on a region of the real world.

17. The electronic device according to claim 16, wherein the control circuit is configured to analyze the image to identify text, and wherein the magnified content includes a magnified version of the text.

18. The electronic device according to claim 16, wherein the control circuit is configured to display an enhancement including computer-generated graphic content based on the gaze point information.

19. The electronic device according to claim 16, wherein the region is characterized by a position within the image of the real world, and wherein the electronic device includes an input-output circuit configured to collect user input to adjust the position.

20. The electronic device according to claim 16, wherein the control circuit is configured to display the magnified content in response to a gaze point gesture.

Images