Image enhancement device with gaze tracking

Abstract

The electronic device may have a display and a camera. The control circuit in the device may collect information about the user's gaze point using a gaze tracking system and other sensors, may collect information about the real-world image by analyzing the real-world image, such as information about content, motion, and other image attributes, may collect user visual information such as user visual acuity, contrast sensitivity, field of view, and geometric distortion, may collect user input such as user preferences and user mode selection commands, and may collect other inputs. Based on the gaze point information and / or other collected information, the control circuit may display the real-world image and supplementary information on the display. The supplementary information may include enhancements such as icons, text labels, and other computer-generated text and graphics overlaid on the real-world image, and may include enhanced image content such as an enlarged portion of the real-world image.

Description

[0001] This patent application claims priority to U.S. Patent Application No. 16 / 280,786 filed on February 20, 2019 and Provisional Patent Application No. 62 / 643,023 filed on March 14, 2018, which are hereby incorporated by reference in their entirety. Background Art

[0002] The present disclosure relates generally to electronic devices, and more particularly to electronic devices that display mixed reality content.

[0003] A mixed reality device may have a camera and a display. The camera may capture an image of the real world. Content such as text and graphics may then be overlaid on the real world image to produce a mixed reality image.

[0004] Presenting mixed reality content to a user can be challenging. Presenting mixed reality images to a user can be disrupted, for example, by head and eye movements. Users can have different visual abilities. If care is not taken, mixed reality images can be presented that cause disorientation or motion sickness and make it difficult for the user to identify items of interest. Summary of the invention

[0005] An electronic device may have a display and a camera. The camera may be used to capture real-world images. The real-world images may be displayed on a display for a user.

[0006] Control circuitry in the device may use gaze tracking systems and other sensors to collect information about the user's gaze point. The control circuitry may also collect information about real-world images by analyzing the real-world images, such as information about content, motion, and other image attributes. User visual information such as user visual acuity, contrast sensitivity, field of view, and geometric distortion may be provided to the control circuitry. Input-output circuitry may be used by the control circuitry to collect user inputs, such as user preferences and user mode selection commands, and may be used to collect position information and other inputs.

[0007] Based on the gaze point information and / or other collected information, the control circuitry may display supplemental information on the real-world image on the display. The supplemental information may include enhancements such as icons, text labels, and other computer-generated text and graphics overlaid on the real-world image, and may include enhanced image content such as an enlarged portion of the real-world image. If desired, the real-world image may be globally or locally enhanced by adjusting contrast, changing image color, enhancing edges, adjusting geometric visual distortion, and performing other image enhancements on the real-world image. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram of an illustrative electronic device according to one embodiment.

[0009] Figure 2 is a diagram of a gaze tracking circuit according to one embodiment.

[0010] Figure 3 is a diagram of an illustrative real-world image and supplemental information displayed on the real-world image according to one embodiment.

[0011] Figure 4 , Figure 5 and Figure 6 is a diagram showing an exemplary image enhancement region on a display according to one embodiment.

[0012] Figure 7 A flow chart of illustrative operations involving providing mixed reality content to a user according to one embodiment. DETAILED DESCRIPTION

[0013] The electronic device may be provided with a display. The display may be used to display supplemental content overlaid on real-world content. The supplemental content may, for example, include enhancements, such as computer-generated text and graphics, and may include enhanced image content, such as real-world image data that has been modified by adjusting image properties, such as contrast, magnification, and other characteristics.

[0014] The electronic device may use information such as information from the gaze tracking system and other information when displaying image enhancement. For example, a portion of the real-world image may be enlarged for a low-vision user only when the user's gaze point is relatively stationary to avoid causing motion sickness effects. The electronic device may analyze the real-world image to identify content of interest (e.g., the location and movement of objects, recognition of faces and other items, the presence of text, the location of the edges of objects, etc.). If necessary, information such as the user's eyeglass prescription and other user visual data may be used to display image enhancement. For example, if the user's vision is impaired (e.g., if the user has low spatial visual acuity), the text in the image may be enlarged, or the image content may be distorted to compensate for the geometric distortion caused by the user's vision. Image enhancement may be applied globally to the real-world image, or may be displayed in an appropriate image enhancement area. If necessary, user input and other information may also be used to display supplemental information on the real-world image.

[0015] Figure 1 1 shows a schematic diagram of an illustrative electronic device of the type that can be used to present mixed reality content to a user. Electronic device 10 may be a head mounted device (head mounted display), a cellular phone, a tablet computer, a head up display (e.g., a mixed reality display in an automobile or other vehicle), a laptop or desktop computer, a television, a watch, or other electronic equipment. Figure 1As shown, the electronic device 10 may have a control circuit 20. The control circuit 20 may include storage and processing circuits for controlling the operation of the device 10. The circuit 12 may include storage devices, such as hard disk drive storage devices, non-volatile memory (e.g., electrically programmable read-only memory configured to form a solid-state drive), volatile memory (e.g., static or dynamic random access memory), etc. The processing circuit in the control circuit 12 may be based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors, power management units, audio chips, graphics processing units, application-specific integrated circuits, and other integrated circuits. Software code may be stored on a storage device in the circuit 20 and run on a processing circuit in the circuit 20 to implement control operations of the device 10 (e.g., operations associated with capturing real-world images with a camera, data acquisition operations, operations involving monitoring the user's gaze point, operations involving generating content overlaid on real-world images, operations associated with performing pattern recognition and other image processing operations on real-world images, operations associated with displaying images with real-world content and overlaid content, etc.).

[0016] Device 10 may include input-output circuitry 22. Input-output circuitry 22 may be used to allow data to be received by device 10 from external equipment (e.g., a computer or other electrical equipment), and to allow a user to provide user input to device 10. Input-output circuitry 22 may also be used to gather information about the environment in which device 10 is operating. Output components in circuitry 22 may allow device 10 to provide output to a user, and may be used to communicate with external electrical equipment.

[0017] like Figure 1 As shown, input-output circuitry 22 may include a camera such as camera 40. Camera 40, which may sometimes be referred to as a front-facing camera, a scene camera, or a real-world image camera, may be used to capture images of the real world (e.g., a user's environment). The real-world images captured by camera 40 may include content in the user's field of view and content outside the user's field of view (e.g., portions of the user's environment adjacent to the user's field of view).

[0018] Input-output circuitry 22 may also include a display, such as display 14. Display 14 may be used to display images to a user of device 10. Display 14 may be an organic light emitting diode display, a liquid crystal display, a liquid crystal on silicon display, a micromirror array display (e.g., a micro-electromechanical system (MEMS) display, sometimes referred to as a digital micromirror device), or any other suitable display. Display 14 may be mounted in device 10 so that the image on display 14 is provided in the user's field of view. In some configurations, display 14 may be supported on a head-mounted support structure (glasses, helmets, hats, goggles, etc.). In other configurations, display 14 may be mounted in a vehicle or other equipment (e.g., in a position where the image may be presented to the user by reflecting the image output from the display from the inner surface of the vehicle window into the user's field of view).

[0019] Display 14 may be located directly in the user's line of sight, or a light combiner in the user's line of sight may be used to merge the image from display 14 with the user's observation of the environment surrounding device 10. In some arrangements, display 14 is housed in a housing of a cellular phone, tablet computer, or other portable electronic device. In general, display 14 may be placed in any suitable location for providing images to a user. During operation, a user may view real-world image content presented through display 14 (e.g., a real-world image captured with a camera and displayed on display 14) and / or may view real-world content directly (e.g., when the real world is visible through a light combiner being used to present an image from display 14 to the user). Exemplary configurations in which a user views real-world content on display 14 overlaid with supplemental content on display 14 are sometimes described herein as examples.

[0020] Input-output circuit 22 may include a gaze tracking system, such as gaze tracking system 16. Gaze tracking (eye monitoring) system 16 may include image sensors, light sources, and / or other equipment for monitoring the user's eyes. Gaze tracking system 16 may include, for example, one or more visible and / or infrared cameras that face the user's eyes and capture images of the user's eyes. During operation of device 10, control circuit 20 may use gaze tracking system 16 to track the user's gaze. Cameras and / or other sensors in system 16 may, for example, determine the position of the user's eyes (e.g., the center of the user's pupil), and may determine the direction of the user's eye orientation (the direction of the user's gaze, sometimes referred to as the measured gaze point). By processing this information and information about the position of display 14, the position of the user's gaze point may be dynamically determined. If desired, additional information (e.g., information from an inertial measurement unit including an accelerometer, compass, and / or gyroscope and / or information from other motion sensors) may be used to determine the position of the user's gaze point. For example, an inertial measurement unit, visual odometry sensors in circuitry 22, and / or other sensors may determine the orientation of device 10 and / or the user's head relative to the surrounding environment and use this information in addition to gaze tracking information to determine the user's current gaze point.

[0021] Sensors and other input devices in the input-output device 18 may also be used to collect user input and other information. The input-output device 18 may include, for example, position and motion sensors (e.g., compasses, gyroscopes, accelerometers, and / or other devices for monitoring the position, orientation, and movement of the device 10), may include force sensors, temperature sensors, touch sensors, buttons, capacitive proximity sensors, light-based proximity sensors, other proximity sensors, color ambient light sensors and other ambient light sensors, strain gauges, gas sensors, pressure sensors, humidity sensors, magnetic sensors, gesture sensors, depth sensors (e.g., three-dimensional structure light sensors and other depth sensors), and other sensors, may include audio components (such as microphones) for collecting voice commands and other audio inputs, and may include speakers for providing audio output (e.g., for providing sound to the user's left and right ears). If desired, the input-output device 18 may include a tactile output device (e.g., a vibration component), light emitting diodes, lasers and other light sources, and other output components.

[0022] Circuitry 22 may include wired and wireless communication circuitry that allows device 10 (e.g., control circuitry 20) to communicate with external equipment (e.g., remote controls, joysticks and other input controllers, portable electronic devices, computers, displays, etc.) and to transmit signals between components (circuits) at different locations in device 10. Global Positioning System (GPS) receiver circuitry and / or other satellite navigation system circuitry in input-output circuitry 22 and / or other position sensor circuitry in circuitry 22 may be used to determine the user's location and / or speed. Using wireless communication circuitry in circuitry 22, device 10 may collect information (e.g., weather information, traffic conditions, etc.) over a network such as the Internet.

[0023] Figure 2 is a diagram showing how the gaze tracking system 16 may collect eye position information about the user's eyes 30. In a typical case, the gaze tracking system 16 may include a system for simultaneously monitoring two eyes (such as Figure 2 Components of the eye 30).

[0024] like Figure 2 As shown, system 16 may include one or more image sensors, such as gaze tracking cameras 24. Each camera 24 may be focused in front of a user's eye, such as eye 30, so that features of the user's eye can be measured. One or more light emitting diodes, lasers, lamps, and / or other light emitting components may be used to form a light source for gaze tracking system 16 (see, e.g., light source 26).

[0025] During operation, light source 26 may emit light 28. For example, light source 26 may emit multiple light beams 28 (e.g., 2-10 light beams, 6 light beams, 4-7 light beams, 4-9 light beams, etc.) toward eye 30 of the user. Light 28 may be reflected off the surface of eye 30. Camera 24 may capture an image of eye 30. Control circuit 20 may use information about the appearance of eye 30 (e.g., iris information, pupil information, blood vessel information, etc.) and / or information about reflected light (e.g., one or more light beams) from cornea 32 and other parts of eye 30 to determine the location of pupil center 36 of pupil 34 and the direction of the user's current gaze (gaze direction 38). Control circuit 20 may use eye position information (pupil center information, eye orientation, etc.) collected by gaze tracking system 16, resulting information determined on the user's gaze direction 38, and information about the location of real-world objects relative to system 16 and eye 30 (e.g., items in the environment surrounding device 10) to dynamically identify the user's gaze point within the real-world environment surrounding device 10. The real-world image may then be augmented and / or enhanced based on the gaze point (eg, the location in the real world where the user is looking).

[0026] During operation, control circuitry 20 may use camera 40 to capture images of the real world, and may use gaze tracking system 16 and other input-output devices 18 to collect user input and information about the user's environment. Data about the user may also be collected, such as the user's eyeglass prescription and other information about the user (e.g., visual impairment details, such as low vision details). This information may be used by control circuitry 20 to display mixed reality content for the user on display 14. For example, text in a real-world image may be enlarged to accommodate a low-vision user. Contrast and other image properties may also be adjusted.

[0027] If desired, the control circuitry 20 may obtain content attributes, such as the identities of people in the real-world image (e.g., by performing pattern recognition operations). Mixed reality content based on these content attributes may be presented to the user. For example, the names of people in the user's field of view may be presented to the user by overlaying text boxes on the real-world image. A text box containing the name of each person in the user's field of view may be placed adjacent to the person, or only those people close to the user's gaze point may be marked. Gaze point information may be used to determine which people the user is observing.

[0028] Gaze point information (e.g., information about the location and motion of the user's gaze point) may also be analyzed to determine when supplemental content should be presented based on real-world image content outside the user's field of view (or only in the user's extreme peripheral vision). For example, if the user's gaze point is moving to the left, the user may be presented with an icon indicating that content of interest (e.g., a moving vehicle) is just to the left of the user's field of view. In this way, the user may be provided with advance notice of an object that is not currently being viewed by the user.

[0029] For example, consider Figure 3 4. The real-world image 50 may be captured by the input-output circuitry 22 using the camera 40. If desired, the field of view of the camera 40 may be larger than the field of view of the user (see, for example, the user's field of view 52). The image 50 may include real-world objects, such as objects 58, 56, and 54 (e.g., people, vehicles, buildings, furniture, landscape elements, etc.). During operation of the device 10, the mixed reality content may be presented to the user in some or all of the user's field of view. For example, computer-generated images (still and / or moving image content with labels and other text and / or graphics) and / or portions of the real-world image 50 that have been enhanced (e.g., by image processing to adjust magnification, contrast, color, brightness, edge detail, etc.) may be presented in one or more areas, such as the supplemental image area 60 (sometimes referred to as a supplemental content area, an overlay area, etc.).

[0030] For example, the information presented in region 60 may include a portion of real-world image 50 that has been enhanced by image processing. The portion of real-world image 50 selected for image enhancement may be selected based on user input and other information. For example, gaze point information may be used to determine which portion of real-world image 50 will be magnified or otherwise enhanced for display in region 60. For example, if a user is viewing object 58 and the user's gaze point overlaps object 58 (see, e.g., gaze point 62), a portion of real-world image 50 associated with some or all of object 58 may be enhanced (e.g., magnified, with additional contrast, etc.), and the resulting enhanced image may be displayed in region 60. If desired, when the user's gaze point does not overlap object 58, but is within a predetermined distance of object 58, some or all of the portion of the image covering object 58 may be enhanced. For example, control circuitry 20 may display enhanced image content in response to detecting that the user's gaze point is within region 66 adjacent to object 62 (see, e.g., gaze point 64).

[0031] When determining when the user's gaze point overlaps with a desired object or an area near a desired object, the control circuit 20 may evaluate how long the user's gaze stays at different locations in the image. A threshold gaze point dwell time or other suitable criteria may be applied to the gaze point information to determine whether the user's gaze point has stayed in a particular location for a long enough time to trigger the display of content in area 60. For example, in response to detecting that the user's gaze point has stayed within area 66 for more than at least 0.2 seconds, at least 0.5 seconds, at least 1 second, at least 2 seconds, at least 4 seconds, at least 10 seconds, less than 8 seconds, less than 5 seconds, or other suitable time, an enlarged version of object 62 may be presented in area 60.

[0032] In some configurations, the control circuit 20 monitors the movement of the user's gaze point and takes action based on the gaze point movement. Figure 3In the example of FIG. 5 , if the user's gaze point 68 is moving in direction 70 toward left edge 72 of the user's field of view 52 (e.g., moving at a speed exceeding a predetermined threshold speed for a predetermined threshold amount of time), the control circuit 20 may analyze the image 50 for objects of interest located outside the field of view 52 (e.g., items located outside left edge 72, such as object 54). A pattern recognition operation may be performed on the image 50 to determine which elements of the image 50 correspond to objects of interest. For example, the image 50 may be processed by the control circuit 20 to identify vehicles, people, objects moving above a certain speed, objects having a brightness exceeding a predetermined brightness threshold, objects of a certain color, objects containing text, etc. In response to determining that the user's gaze point 68 is moving in direction 70 and an item of interest in the image 50, such as object 54, is located outside the user's current field of view 52 in direction 70, associated supplemental information may be presented to the user. For example, the control circuit 20 may display a graphical element such as an arrow 74 pointing to an out-of-view object such as object 54. This directs the user's attention to potentially important objects just beyond the user's field of view and helps the user keep an eye on such objects. For example, if the user is driving a vehicle or walking along a street, it may be helpful to be notified of the presence of an object 54 (e.g., a fast-moving vehicle). In some arrangements, disorientation and motion sickness may be avoided by suppressing the display of enlarged text or other supplemental content in response to detecting movement of the user's gaze point (e.g., eye saccades).

[0033] Any suitable criteria may be used to determine when the control circuit 20 should display supplemental information (e.g., overlay content such as enhanced image content in area 60 and / or text or graphics such as arrow 74, text labels, warnings, etc.) on the real-world image. For example, the control circuit 20 may determine the location of the user's gaze point and the movement of the user's gaze point by processing gaze point location information from the gaze tracking system 16 and information about the orientation of the user's head from a position and motion sensor (orientation sensor) such as an inertial measurement unit and / or a visual odometry system. If the location and / or movement of the gaze point meets predetermined criteria (e.g., if the gaze point stays at a particular location for more than a predetermined amount of time, if the gaze point moves at a particular speed and / or direction), appropriate supplemental information may be displayed. As another example, supplemental information may be displayed based on information such as the location of device 10 (e.g., the location of device 10 determined by a satellite navigation system receiver and / or other location sensor in input-output device 18), sensor information (e.g., information about ambient lighting conditions indicating whether it is daytime or nighttime), based on the content in image 50 (e.g., whether a face is present and / or whether the face present corresponds to a particular individual, whether image 50 contains a particular object, is characterized by a predetermined amount of motion, has a particular brightness level, or contains a particular color or pattern, etc.), and / or based on other information. Display of supplemental information may also be suppressed (e.g., temporarily) whenever certain criteria are met (e.g., excessive motion of the user's gaze point, etc.).

[0034] If desired, the user of device 10 may press a button, speak a voice command, orient the user's gaze in a particular direction (e.g., gaze downward to invoke auto-bifocus, gaze upward and to the right, gaze so that the user's gaze point overlaps with a displayed interactive icon, etc.), blink (e.g., blink three times in rapid succession so that camera 24 or other sensor can detect a predetermined blink command), or may otherwise take action to direct control circuit 20 to display supplemental information.

[0035] The conditions that determine whether to display supplemental information and the type of supplemental information presented (e.g., magnified image content, contrast enhanced image content, etc.) may include user data, such as the user's eyeglass prescription and other visual function information. For example, the user's vision may be characterized using ophthalmic instruments (e.g., to characterize the user's field of view, visual acuity, contrast sensitivity, and geometric distortion). This information may then be used to determine, for example, whether the image content should be enhanced by increasing contrast or by increasing magnification (or both). In some cases (e.g., when the user's gaze point is moving), motion sickness may be induced if magnified content is presented. Therefore, the presentation of supplemental information (such as enhanced image information in area 60) may be suppressed when gaze point motion exceeding a predetermined amount is detected or other suitable supplemental information suppression criteria are met.

[0036] Supplemental information (e.g., computer-generated text or graphics or other enhanced information and / or enhanced image content) may be displayed in any suitable portion of the user's field of view. For example, the supplemental information may be displayed on the Figure 4 region 60 (eg, the middle of the displayed image), Figure 5 region 60 (e.g., along an edge of the displayed image, such as along a bottom edge of the displayed image), Figure 6 region 60 (eg, in the upper right corner of the displayed image or other corner of the displayed image), and / or in other suitable portions of the image being displayed by display 14.

[0037] Figure 7 A flow chart illustrating illustrative operations involving using device 10 to provide real-world images and supplemental information to a user on display 14 is shown.

[0038] During the operation of block 80, control circuitry 20 may use input-output circuitry 22 to collect information about the user's gaze. Gaze tracking system 16 and other sensors in the inertial measurement unit and / or input-output device 18 may be used to monitor eye movement and optionally other inputs such as head movement / orientation and / or information about the orientation of device 10, and thereby measure the user's gaze point. Information about the user's gaze point (such as gaze point position and / or gaze point movement) may be analyzed to determine whether the user is providing intentional input (looking at a predetermined area of ​​the displayed image for a predetermined time to intentionally invoke the display of supplemental information), and / or information about the user's gaze point (such as gaze point position and / or gaze point movement) may be analyzed to determine whether the user is looking at a particular item of interest or moving toward a particular item of interest (as an example). The operation of block 80 may include characterizing the user's eye movement as fixation (not moving), saccade (moving), and blinking.

[0039] During operation of block 82, control circuitry 20 may analyze captured images, such as image 50. For example, an image of a scene being viewed by a user (e.g., real-world image 50) may be processed using image recognition algorithms and other processing algorithms to identify content and other attributes in the image. For example, an image may be processed to identify image attributes such as brightness, color breakdown, contrast, etc. Patterns may be detected (e.g., to identify vehicles, roads, people, faces, identities of people associated with particular faces, road signs with or without text, text on billboards and other advertisements and / or other items in the user's environment). Edge detection processes may be used to detect object edges. Stationary objects and moving objects may be classified. Techniques such as these may be used to process all of the information in image 50, or a subset of the information in image 50 may be processed if desired. For example, image processing such as facial recognition processing may be performed for those faces on which the user's gaze rests, but the image processing may be skipped for other faces to reduce processing load. The processing of this information may also be affected by the output of other sensors. For example, road sign information may be processed and identified for magnification only when the user and device 10 are moving faster than a predetermined speed (indicating that the user is driving).

[0040] During the operation of block 84, information about vision and other attributes of the user may be collected. For example, user data such as visual impairment information may be collected using an ophthalmic instrument and provided to the control circuitry 20 (e.g., via a wired and / or wireless communication path, using a user input device, etc.). By characterizing the user's visual acuity, contrast sensitivity, field of view, geometric distortion, and other visual attributes, appropriately targeted image enhancement and intensification may be provided for the user.

[0041] During the operation of block 86, user input may be collected using input-output circuitry 22. For example, gaze tracking system 16 may collect intentional eye input such as blinking and intentional gaze point input (e.g., eye gestures, such as gestures involving moving the user's gaze point in and out of a particular area or moving the user's gaze in an intentional direction downward to automatically invoke a strip of magnified image 50 for use as digital bifocals, etc.). Circuitry 22 may also collect user input using buttons, touch sensors, microphones, and other input-output circuitry. User input may be used to place device 10 in different operating modes (e.g., a walking mode in which supplemental information is provided in the center of the displayed image; a driving mode in which supplemental information is provided along the edge of the displayed image; a meeting mode in which facial recognition operations are used to identify and tag participants in a meeting; an inactive mode in which no supplemental information is displayed, etc.). User input may also be used to provide preferences for device 10 (e.g., user-adjustable settings such as preferred display brightness levels, thresholds, magnification settings, desired locations of supplemental information regions 60, etc.).

[0042] Additional inputs may be collected by input-output circuitry 22 during the operations of block 88. For example, control circuitry 20 may use circuitry 22 to collect location information, information regarding the speed and direction of motion of device 10, information regarding traffic conditions, information regarding local weather, and the like.

[0043] During operation of block 90, control circuitry 20 may display images captured from camera 40 on display 14 for viewing by a user (and / or the user may view the real world through a light combiner such as a car window or a light combiner in a head mounted device). Supplemental information may be displayed on display 14 to overlap with real world content. For example, information may be displayed on some or all of display 14 that includes enhanced visual information such as graphics, text, and other computer generated overlays and / or includes enhanced visual information such as all or some of image 50 that has been processed to adjust contrast, to adjust brightness, to adjust color, to adjust object features to enhance edges by applying edge enhancement, to adjust magnification, to adjust geometric distortion, or to adjust other image attributes. User data collected during operation of block 84 (e.g., details of the user's visual impairment) may be considered during operation of block 90. ​​For example, if the user's vision is characterized by geometric distortion, compensating geometric distortion may be applied when displaying images on display 14. If the user has low visual spatial acuity, the content may be appropriately magnified.

[0044] In general, the supplemental information may be displayed based on information collected during block 80 (e.g., gaze point information), image analysis performed during block 82, information collected during block 84, information collected during block 86 (e.g., mode selection information and other information provided as user input), and / or information collected during operation at block 88 (e.g., information about current weather, traffic conditions, etc.). The image may be locally adjusted (e.g., zooming in on an appropriate area such as Figure 4 , Figure 5 and Figure 6 The contrast ratio may be applied to the text in region 60 of the image (eg, by enhancing the contrast of the entire displayed image when it is determined that the user's vision is characterized by low contrast).

[0045] According to one embodiment, an electronic device is provided, comprising: a camera configured to capture images of the real world; a display configured to display the images; a gaze tracking system configured to collect gaze point information; and a control circuit configured to display enlarged supplementary content in the image on the display based on the gaze point information.

[0046] According to another embodiment, the magnified supplemental content includes an enlarged portion of the image of the real world.

[0047] According to another embodiment, the control circuit is configured to display the enlarged portion of the image of the real world as an overlay on an area of ​​the image of the real world.

[0048] According to another embodiment, the region is characterized by a location within the image of the real world, and the electronic device includes input-output circuitry configured to capture user input to adjust the location.

[0049] According to another embodiment, the control circuit is configured to display the magnified portion in response to a gaze point gesture.

[0050] According to another embodiment, the control circuit is configured to display the enlarged supplemental content in response to a user blink input captured using the gaze tracking system.

[0051] According to another embodiment, the control circuit is configured to display the enlarged supplemental content in response to determining from the gaze point information that a gaze point has remained on a portion of the image for more than a predetermined gaze point dwell time.

[0052] According to another embodiment, the control circuit is configured to display an enhancement including computer-generated graphical content in the image on the display based on the gaze point information.

[0053] According to another embodiment, the control circuit is configured to display the magnified supplemental content based on information about vision attributes of the user.

[0054] According to another embodiment, the control circuit is configured to analyze the images to identify objects in motion.

[0055] According to another embodiment, the control circuit is configured to analyze the image to identify a face, and the control circuit is configured to display text that labels the identified face with an identity associated with the identified face.

[0056] According to another embodiment, the control circuit is configured to analyze the image to identify text, and the enlarged supplemental content includes an enlarged version of the text.

[0057] According to another embodiment, the electronic device includes input-output circuitry, the control circuitry is configured to collect user input using the input-output circuitry, and the control circuitry is configured to display the enlarged supplemental content based on an operating mode selected using the user input.

[0058] According to another embodiment, the input-output circuit includes a position sensing circuit, and the control circuit is configured to display the enlarged supplemental content based on a geographic location identified using the position sensing circuit.

[0059] According to another embodiment, the control circuit is configured to provide the amplified supplemental information based on a gaze point dwell time associated with the gaze point information, attributes in the image, user vision attributes, and user input.

[0060] According to another embodiment, the electronic device includes a motion sensor, and the control circuit is configured to display the enlarged supplemental content in response to determining that the speed measured by the motion sensor exceeds a predetermined limit.

[0061] According to one embodiment, an electronic device is provided, comprising: a camera configured to capture images of the real world; a display; a gaze tracking system configured to collect gaze point information; and a control circuit configured to display enhanced image content overlapping the image on the display based on the gaze point information.

[0062] According to another embodiment, the control circuit is configured to collect user visual information including user visual characteristics, wherein the user visual characteristics are selected from the group consisting of: visual acuity, contrast, field of view and geometric distortion, and the control circuit is configured to display the enhanced image content based on the collected user visual information.

[0063] According to another embodiment, the enhanced image content includes an enhanced portion of the image, and the enhanced portion of the image is selected from the group consisting of: an enlarged portion of the image; a portion of the image with increased contrast, a portion of the image with enhanced edges, and a portion of the image with changed color.

[0064] According to one embodiment, an electronic device is provided, comprising: a camera configured to capture an image of the real world; a gaze tracking circuit configured to collect gaze point information; a display; and a control circuit configured to: analyze the image of the real world to identify text in the image; and display an enlarged version of the text overlaid on a portion of the image on the display based on the gaze point information.

[0065] The foregoing is merely illustrative and various modifications may be made to the embodiments described. The foregoing embodiments may be implemented independently or in any combination.

Claims

1. An electronic device, comprising: a camera configured to capture images of the real world; a display configured to display the image; A gaze tracking system, wherein the gaze tracking system is configured to collect gaze point information; and A control circuit is configured to display enlarged supplementary content in the image on the display based on the gaze point information, wherein the gaze point information includes a position and a movement of a gaze point, and wherein the control circuit displays the enlarged supplementary content based on the position and the movement of the gaze point.

2. The electronic device of claim 1 , wherein the magnified supplemental content comprises an enlarged portion of the image of the real world, and wherein the control circuitry is configured to display the enlarged portion of the image of the real world as an overlay on an area of ​​the image of the real world.

3. An electronic device according to claim 2, wherein the area is characterized by a position within the image of the real world, and wherein the electronic device includes input-output circuitry configured to collect user input to adjust the position. 4 . The electronic device of claim 2 , wherein the control circuit is configured to display the magnified portion in response to a gaze point gesture. 5 . The electronic device of claim 1 , wherein the control circuit is configured to display the enlarged supplemental content in response to a user blink input collected using the gaze tracking system.

6. An electronic device according to claim 1, wherein the control circuit is configured to display the enlarged supplemental content in response to determining from the gaze point information that the gaze point has stayed on a portion of the image for more than a predetermined gaze point stay time.

7. The electronic device of claim 1, wherein the control circuit is configured to display an enhancement including computer-generated graphical content in the image on the display based on the gaze point information. 8 . The electronic device of claim 1 , wherein the control circuit is configured to display the magnified supplemental content based on information about a user's vision attributes.

9. The electronic device of claim 1, wherein the control circuit is configured to analyze the image to identify an object in motion.

10. The electronic device of claim 1, wherein the control circuitry is configured to analyze the image to identify a face, and wherein the control circuitry is configured to display text that labels the identified face with an identity associated with the identified face.

11. The electronic device of claim 1, wherein the control circuit is configured to analyze the image to identify text, and wherein the enlarged supplemental content comprises an enlarged version of the text.

12. The electronic device of claim 1 , further comprising an input-output circuit, wherein the control circuit is configured to collect user input using the input-output circuit, and wherein the control circuit is configured to display the enlarged supplemental content based on an operating mode selected using the user input.

13. The electronic device of claim 12, wherein the input-output circuitry includes position sensing circuitry, and wherein the control circuitry is configured to display the enlarged supplemental content based on a geographic location identified using the position sensing circuitry.

14. The electronic device of claim 1, wherein the control circuit displays enlarged supplemental content based on a direction of the movement of the gaze point.

15. The electronic device of claim 1, further comprising a motion sensor, wherein the control circuit is configured to display the enlarged supplemental content in response to determining that the speed measured by the motion sensor exceeds a predetermined limit.

16. An electronic device, comprising: a camera configured to capture images of the real world; monitor; A gaze tracking system, wherein the gaze tracking system is configured to collect gaze point information; and A control circuit is configured to identify a user's vision attributes and to display enhanced image content overlapping the image on the display based on the gaze point information and the user's vision attributes.

17. The electronic device of claim 16, wherein the user vision attribute is selected from the group consisting of: user eyeglass prescription, user field of view, user visual acuity, user contrast sensitivity, and user geometric distortion.

18. An electronic device according to claim 17, wherein the enhanced image content includes an enhanced portion of the image, and the enhanced portion of the image is selected from the group consisting of: an enlarged portion of the image, a portion of the image with increased contrast, a portion of the image with enhanced edges, and a portion of the image with changed color.

19. The electronic device of claim 16, wherein the gaze point information comprises movement of the gaze point, and wherein the control circuit is configured to display the enhanced image content based on the movement of the gaze point not exceeding a predetermined amount.

20. An electronic device, comprising: a camera configured to capture images of the real world; A gaze tracking circuit, wherein the gaze tracking circuit is configured to collect gaze point information; monitor; and A control circuit, the control circuit being configured to: analyzing the image of the real world to recognize text in the image; displaying, on the display, an enlarged version of the text overlaid on a portion of the image based on the gaze point information; as well as Displaying the enlarged version of the text is ceased at least in part in response to detecting motion in the gaze point information.

Images