Head-mounted device with diplopia compensation and improved converging and diverging comfort

Abstract

The invention relates to a head-mounted device with diplopia compensation and cohesive comfort improvement. A method of operating an electronic device to mitigate diplopia is provided. The method may include acquiring unvarnished content using a front-facing camera, generating virtual content using a graphics renderer, mitigating diplopia or improving cohesive comfort by at least modifying the unvarnished content, merging the unvarnished content with the virtual content, and displaying the merged content. The pass-through content may be modified by selectively shifting or transforming the left-eye pass-through content or the right-eye pass-through content. The virtual content may be rendered based on similar types of modifications applied to the unvarnished content. The virtual content may also be modified by selectively shifting or transforming the left-eye virtual content or the right-eye virtual content.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 740,287 filed on June 11, 2024 and U.S. Provisional Patent Application No. 63 / 590,552 filed on October 16, 2023, each of which is hereby incorporated by reference in its entirety. Technical Field

[0002] The present disclosure relates generally to electronic devices, and more particularly to electronic devices such as head mounted devices. Background Art

[0003] An electronic device such as a head-mounted device may have a camera for capturing a video feed of an external scene, an image rendering unit for generating virtual content, and one or more displays for presenting the captured video feed of the scene and / or the generated virtual content to a user.

[0004] Designing a head-mounted device for a user who wears prescription glasses can be challenging. Most standard eyeglass prescriptions, including those that require prismatic correction, can be adapted for use with an additional clip-on lens in front of the display. Prismatic correction is a special type of eyeglass prescription that helps people with double vision by aligning the images seen by both eyes. Double vision occurs when the eyes do not work together correctly, causing the eyes to see two separate images of a single object. Prismatic correction can also be used to provide improved vergence comfort, even for users without double vision. However, the use of clip-on lenses can be unnecessarily cumbersome and can reduce the accuracy of and limit the range of eye tracking sensors in the head-mounted device. Summary of the invention

[0005] An electronic device such as a head mounted device may include a graphics rendering unit for generating virtual content and one or more displays configured to output computer generated virtual content. The electronic device may optionally include one or more cameras configured to capture a video feed of a real world environment facing the camera. The captured video feed (sometimes referred to as a pass-through feed or pass-through content) may optionally be presented on one or more displays and / or merged with the computer generated virtual content before being displayed.

[0006] One aspect of the present disclosure provides a method for operating an electronic device such as a head-mounted device, the method comprising: acquiring transparent content using one or more image sensors, generating virtual content using a graphics rendering subsystem, reducing diplopia or improving vergence comfort by modifying the transparent content to generate corresponding modified transparent content, merging the modified transparent content with virtual content to generate corresponding merged content, and presenting the merged content using one or more displays. The transparent content can be modified by cropping or transforming the left eye transparent content or the right eye transparent content at an image signal processor. The transparent content can be modified by distorting the left eye transparent content or the right eye transparent content to apply a lateral offset in the field of view or a transformation in the viewpoint.

[0007] The graphics rendering subsystem may be configured to generate virtual content based on a first virtual camera and a second virtual camera, the first virtual camera applying a constraint that defines a first viewpoint from which the left eye virtual content is rendered at the graphics rendering subsystem, and the second virtual camera applying a constraint that defines a second viewpoint from which the right eye virtual content is rendered at the graphics rendering subsystem. The first virtual camera may have a first view cone, and the second virtual camera may have a second view cone. The graphics rendering subsystem may be configured to shift the left eye virtual content or the right eye virtual content on one or more displays by adjusting the angular limits (boundaries) of the corresponding view cones in the first view cone and the second view cone. The graphics rendering subsystem may be configured to shift the selected view cone in the first view cone and the second view cone laterally along a common projection plane. The graphics rendering subsystem may be configured to transform the selected virtual camera in the first virtual camera and the second virtual camera. The virtual content may be modified by selectively distorting the left eye virtual content or the right eye virtual content to apply a lateral shift in the field of view or a transformation in the viewpoint.

[0008] One aspect of the present disclosure provides a method for operating a device, the method comprising: acquiring transparent content using one or more image sensors, generating virtual content using a graphics rendering subsystem, obtaining a prism prescription value for correcting diplopia or improving vergence comfort, calculating compensation parameters based on the prism prescription values, modifying the transparent content based on the compensation parameters, and displaying a mixed version of the modified transparent content and virtual content. The method may also include modifying the transparent content by shifting or transforming the left eye transparent content or the right eye transparent content. The method may also include modifying the virtual content based on the compensation parameters. The method may also include providing the user with an opportunity to adjust the compensation parameters to minimize diplopia or improve vergence comfort. The method may also include reading a scannable code and obtaining a prism prescription value from the scannable code.

[0009] One aspect of the present disclosure provides a method for operating an electronic device, the method comprising: obtaining transparent content, generating virtual content, displaying a stimulus point, monitoring a gaze point while the stimulus point is being displayed to estimate a misalignment between the gaze point and an expected gaze point, calculating a compensation parameter based on the estimated misalignment, modifying the transparent content based on the compensation parameter, and displaying a hybrid version of the modified transparent content and the virtual content. The method may also include: estimating the misalignment by obtaining a plurality of gaze data points, removing one or more outliers from the plurality of gaze data points, and averaging the plurality of gaze data points.

[0010] One aspect of the present disclosure provides a method, which includes: generating virtual content having left-eye content and right-eye content using a graphics rendering subsystem, the graphics rendering subsystem being configured to generate the virtual content based on a first virtual camera and a second virtual camera, the first virtual camera applying a constraint that defines a first viewpoint from which the left-eye content is rendered at the graphics rendering subsystem, and the second virtual camera applying a constraint that defines a second viewpoint from which the right-eye content is rendered at the graphics rendering subsystem; alleviating diplopia or improving vergence comfort by modifying a selected one of the left-eye content and the right-eye content to generate the modified virtual content; and displaying the modified content. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a top view of an illustrative head mounted device according to some embodiments.

[0012] Figure 2 is a schematic diagram of an illustrative electronic device according to some embodiments.

[0013] Figure 3 is a diagram illustrating illustrative hardware and / or software subsystems within an electronic device configured to perform diplopia compensation according to some embodiments.

[0014] Figure 4 is a diagram illustrating shifting of projected content to be displayed to achieve double vision correction according to some embodiments.

[0015] Figure 5 is a diagram illustrating camera perspective transformation to achieve double vision correction according to some embodiments.

[0016] Figure 6 According to some embodiments, a method for operating a combination of Figures 1 to 5 A flow chart of illustrative steps for an electronic device of the type shown.

[0017] Figure 7 is a flow chart of illustrative steps for generating diplopia compensation parameters based on eyeglass prescription values ​​according to some embodiments.

[0018] Figure 8 is a flow chart of illustrative steps for generating diplopia compensation parameters based on measured gaze data according to some embodiments. DETAILED DESCRIPTION

[0019] Figure 1 A top view of an exemplary head mounted device is shown in FIG. Figure 1 As shown in , a head-mounted device such as electronic device 10 may have a head-mounted support structure such as housing 12. Housing 12 may include a portion (e.g., head-mounted support structure 12T) for allowing device 10 to be worn on the user's head. Support structure 12T may be formed of fabric, polymer, metal, and / or other materials. Support structure 12T may form a strap or other head-mounted support structure to help support device 10 on the user's head. The main support structure of housing 12 (e.g., a head-mounted housing such as main housing portion 12M) may support electronic components such as display 14.

[0020] The main housing portion 12M may include a housing structure formed of metal, polymer, glass, ceramic, and / or other materials. For example, the housing portion 12M may have a housing wall on the front side F and housing walls on adjacent top, bottom, left, and right sides formed of a rigid polymer or other rigid support structure, and these rigid walls may be optionally covered with electronic components, fabric, leather, or other soft materials, etc. The housing portion 12M may also have an internal support structure, such as a frame (chassis) and / or a structure that performs multiple functions (such as controlling airflow and heat dissipation while providing structural support).

[0021] The walls of housing portion 1254 may enclose internal components 38 in interior region 34 of device 10, and may separate interior region 34 from the environment (exterior region 36) surrounding device 10. Internal components 38 may include integrated circuits, actuators, batteries, sensors, and / or other circuits and structures for device 10. Housing 12 may be configured to be worn on a user's head, and may form glasses, spectacles, a hat, a mask, a helmet, goggles, and / or other head-mounted devices. Configurations in which housing 12 forms goggles are sometimes described herein as examples.

[0022] The front side F of the housing 12 may face outwardly, away from the user's head and face. The opposite rear side R of the housing 12 may face the user. The portion of the housing 12 located on the rear side R (e.g., the portion of the main housing 12M) may form a cover, such as cover 12C (sometimes referred to as a blind). The presence of cover 12C on the rear side R may help hide the internal housing structure, the interior components 38, and other structures in the interior area 34 from being seen by the user.

[0023] Device 10 may have one or more cameras, such as Figure 1Camera 46. Camera 46 mounted on front face F and facing outward (toward the front of device 10 and away from the user) may sometimes be referred to herein as a forward-pointing or front-facing camera. Camera 46 may capture visual odometry information, image information processed to locate objects in the user's field of view (e.g., so that virtual content can be properly aligned with respect to real-world objects), image content displayed in real time to the user of device 10, and / or other suitable image data. For example, a forward-pointing (positive) camera may allow device 10 to monitor movement of device 10 relative to the environment surrounding device 10 (e.g., the camera may be used to form part of a visual odometry system or a visual-inertial odometry system). The forward-pointing camera may also be used to capture images of the environment displayed to the user of device 10. If desired, images from multiple forward-pointing cameras may be merged with each other and / or the forward-pointing camera content may be merged with computer-generated content for the user.

[0024] Device 10 may have any suitable number of cameras 46. For example, device 10 may have K cameras, where the value of K is at least one, at least two, at least four, at least six, at least eight, at least ten, at least 12, less than 20, less than 14, less than 12, less than 10, 4-10, or other suitable value. Cameras 46 may be sensitive at infrared wavelengths (e.g., camera 46 may be an infrared camera), may be sensitive at visible wavelengths (e.g., camera 46 may be a visible camera), and / or cameras 46 may be sensitive at other wavelengths. If desired, cameras 46 may be sensitive at both visible and infrared wavelengths.

[0025] The device 10 may have a left optical module and a right optical module 40. The optical module 40 supports electronic components and optical components such as light-emitting components and lenses, and therefore may sometimes be referred to as an optical assembly, an optical system, an optical component support structure, a lens and display support structure, an electronic component support structure, or a housing structure. Each optical module may include a corresponding display 14, a lens 30, and a support structure such as a support structure 32. The support structure 32, which may sometimes be referred to as a lens support structure, an optical component support structure, an optical module support structure, or an optical module portion, or a lens barrel, may include a hollow cylindrical structure with an open end or other support structures for accommodating the display 14 and the lens 30. The support structure 32 may, for example, include a left lens barrel that supports the left display 14 and the left lens 30 and a right lens barrel that supports the right display 14 and the right lens 30.

[0026] Display 14 may include an array of pixels or other display devices to produce images. Display 14 may include, for example, organic light emitting diode pixels formed on a substrate having thin film circuits and / or formed on a semiconductor substrate, pixels formed from a crystalline semiconductor die, liquid crystal display pixels, scanning display devices, and / or other display devices for producing images.

[0027] Lens 30 may include one or more lens elements for providing image light from display 14 to corresponding eye zone 13. The lens may be implemented using a refractive glass lens element, using a mirror lens structure (a catadioptric lens), using a Fresnel lens, using a holographic lens, and / or other lens systems.

[0028] When the user's eyes are located in the eye zone 13, the display (display panel) 14 operates together to form the display of the device 10 (e.g., the user's eyes can view the images provided by the corresponding left and right optical modules 40 in the eye zone 13, so that a stereoscopic image is created for the user). When the user views the display, the left image from the left optical module is fused with the right image from the right optical module.

[0029] It may be desirable to monitor the user's eyes when the user's eyes are located in the eye-adaptive zone 13. For example, it may be desirable to use a camera to capture an image of the user's iris (or other parts of the user's eyes) for user authentication. It may also be desirable to monitor the direction of the user's gaze. Gaze tracking information can be used as a form of user input and / or can be used to determine where the image content resolution should be locally enhanced within the image in the fovea imaging system. In order to ensure that the device 10 can capture a satisfactory eye image when the user's eyes are located in the eye-adaptive zone 13, each optical module 40 may be provided with a camera (such as camera 42) and one or more light sources (such as light emitting diodes 44) or other light emitting devices (such as lasers, lamps, etc.). The camera 42 and the light emitting diode 44 can operate at any suitable wavelength (visible light, infrared light, and / or ultraviolet light). For example, the diode 44 can emit infrared light that is invisible (or almost invisible) to the user. This allows the eye monitoring operation to be performed continuously without interfering with the user's ability to view the image on the display 14.

[0030] Figure 2 A schematic diagram of an illustrative electronic device (such as a head-mounted device or other wearable device) is shown in FIG. Figure 2 Device 10 may operate as a standalone device and / or resources of device 10 may be used to communicate with external electronic equipment. For example, communication circuitry in device 10 may be used to send user input information, sensor information, and / or other information to an external electronic device (e.g., wirelessly or via a wired connection). Each of these external devices may include Figure 2 Components of the type shown in device 10.

[0031] like Figure 2As shown in , a head mounted device (such as device 10) may include a control circuit 20. The control circuit 20 may include storage and processing circuits for supporting the operation of the device 10. The storage and processing circuits may include storage devices such as non-volatile memory (e.g., flash memory or other electrically programmable read-only memory configured to form a solid state drive), volatile memory (e.g., static or dynamic random access memory), etc. One or more processors in the control circuit 20 may be used to collect inputs from sensors and other input devices, and may be used to control output devices. The processing circuit may be based on one or more processors (such as microprocessors, microcontrollers, digital signal processors, baseband processors and other wireless communication circuits, power management units, audio chips, application specific integrated circuits, etc.). During operation, the control circuit 20 may use the display 14 and other output devices to provide visual output and other outputs to the user. The control circuit 20 may be configured to perform operations in the device 10 using hardware (e.g., dedicated hardware or circuits), firmware, and / or software. The software code for performing operations in the device 10 may be stored on a storage circuit (e.g., a non-transitory (tangible) computer-readable storage medium storing software code). The software codes may sometimes be referred to as program instructions, software, data, instructions or code. The stored software codes may be executed by processing circuitry within circuitry 20.

[0032] To support communication between device 10 and external equipment, control circuit 20 may communicate using communication circuit 22. Circuit 22 may include an antenna, radio frequency transceiver circuits, and other wireless communication circuits and / or wired communication circuits. Circuit 22 (which may sometimes be referred to as control circuits and / or control and communication circuits) may support two-way wireless communication between device 10 and external equipment (e.g., companion devices such as computers, cellular phones, or other electronic devices, accessories such as pointing devices, controllers, computer styluses or other input devices, speakers or other output devices, etc.) via a wireless link.

[0033] For example, circuitry 22 may include radio frequency transceiver circuitry, such as wireless local area network transceiver circuitry configured to support communication via a wireless local area network link, near field communication transceiver circuitry configured to support communication via a near field communication link, cellular telephone transceiver circuitry configured to support communication via a cellular telephone link, or transceiver circuitry configured to support communication via any other suitable wired or wireless communication link. link, Wireless communications may be supported by a wireless link, a wireless link operating at frequencies between 10 GHz and 400 GHz, a 60 GHz link or other millimeter wave link, a cellular telephone link, or other wireless communications link. Device 10 may include power circuits for sending and / or receiving wired and / or wireless power, if desired, and may include a battery or other energy storage device. For example, device 10 may include a coil and a rectifier to receive wireless power provided to circuits in device 10.

[0034] Device 10 may include input-output devices such as device 24. Input-output devices 24 may be used to collect user input, to collect information about the user's surroundings, and / or to provide output to the user. Device 24 may include one or more displays such as display 14. Display 14 may include one or more display devices such as an organic light emitting diode display panel (a panel having organic light emitting diode pixels formed on a polymer substrate or silicon substrate containing pixel control circuitry), a liquid crystal display panel, a micro-electromechanical system display (e.g., a two-dimensional mirror array or scanning mirror display device), a display panel having an array of pixels formed from crystalline semiconductor light emitting diode dies (sometimes referred to as micro-LEDs), and / or other display devices.

[0035] Sensors 16 in input-output devices 24 may include force sensors (e.g., strain gauges, capacitive force sensors, resistive force sensors, etc.), audio sensors (such as microphones), touch and / or proximity sensors (such as capacitive sensors, such as touch sensors forming buttons, trackpads, or other input devices), and other sensors. If desired, sensors 16 may include optical sensors (such as optical sensors that emit and detect light), ultrasonic sensors, optical touch sensors, optical proximity sensors, and / or other touch sensors and / or proximity sensors, monochrome and color ambient light sensors, image sensors (e.g., cameras), fingerprint sensors, iris scan sensors, retina scan sensors, and other biometric sensors, temperature sensors, sensors for measuring three-dimensional contactless gestures (“air gestures”), pressure sensors, sensors for detecting the position, orientation, and / or movement of device 10 and / or information about the user's head posture (e.g., accelerometers, magnetic sensors such as compass sensors, etc.), and / or proximity sensors. Sensors 16 and / or other input-output devices may be used to collect user input. For example, a button may be used to collect button press input, a touch sensor overlapped with a display may be used to collect user touch screen input, a touch pad may be used to collect touch input, a microphone may be used to collect audio input (e.g., voice commands), an accelerometer may be used to monitor when a finger contacts an input surface and may therefore be used to collect finger press input, etc.

[0036] If desired, the electronic device 10 may include additional components (see, for example, other devices 18 in input-output devices 24). Additional components may include tactile output devices, actuators for moving the movable housing structure, audio output devices such as speakers, light emitting diodes for status indicators, light sources such as light emitting diodes that illuminate portions of the housing and / or display structure, other optical output devices, and / or other circuits for collecting input and / or providing output. The device 10 may also include a battery or other energy storage device, a connector port for supporting wired communication with auxiliary equipment and for receiving wired power, and other circuits.

[0037] Display 14 may be used to present various content to the user's eyes. When viewed through eye zone 13, left and right displays 14 used to present a synthetic stereoscopic image to the user's eyes may sometimes be collectively referred to as display 14. For example, display 14 may present virtual reality (VR) content. Virtual reality content may refer to content that includes only virtual objects in a virtual reality (computer-generated) environment. As another example, display 14 may present mixed reality (MR) content. Mixed reality content may refer to content that includes virtual objects and real objects from the real-world physical environment in which device 10 is running. As another example, display 14 may only present real-world content. Real-world content may refer to content captured by one or more front-facing cameras (see, e.g., Figure 1 The real-world content captured by the front-facing camera is thus sometimes referred to as a camera pass-through feed, a (live) video pass-through feed, or a pass-through video feed (stream).

[0038] A physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In the case of an XR system, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one law of physics.

[0039] Wearable Figure 1 and Figure 2 Users of electronic devices 10 of the type described may sometimes have a medical condition known as double vision or "diplopia," which is a visual disorder that causes a person to see two different images of a single object. The two different images may appear side by side, vertically stacked, or at various angles relative to each other. Users with double vision may require eyeglass prescriptions with prismatic correction. Prismatic correction typically involves the use of specially designed eyeglass lenses that incorporate prisms configured to adjust the path of light entering the eye in order to reduce or eliminate double vision. Thus, prismatic correction techniques shift image content as seen by the user in a particular direction to enable the user to fuse the two different images as seen by their eyes. Providing prismatic correction in a head mounted device can be challenging.

[0040] According to one embodiment, the head mounted device 10 may be provided with software and / or hardware subsystems configured to perform diplopia compensation / mitigation and provide improved vergence comfort without the use of additional clip-on lenses. Figure 3 This type of electronic device 10 is illustrated in FIG. Figure 3 As shown, device 10 may include one or more image sensors such as image sensor 50, an image signal processor such as image signal processor (ISP) 52, one or more tracking sensors such as tracking sensor 54, one or more processing (computing) units such as central processing unit (CPU) 60, neural processing unit (NPU) 56 and graphics processing unit (GPU) 70, a warping subsystem such as warp block 80, a media merging subsystem such as media merging synthesizer 90, and one or more displays such as display 14.

[0041] Sensor 50 may include one or more front cameras and / or other cameras for collecting information about the external real-world environment around device 10. Camera 50 may represent Figure 1 One or more front cameras 46 in. The video feed output from the camera 50 may sometimes be referred to as a raw video feed or a real-time video pass-through feed or a real-time pass-through video stream. Therefore, the front camera 50 is sometimes referred to as a pass-through camera in this article. This pass-through feed output from the camera 50 may be processed by an image signal processor (ISP) 52 configured to perform image signal processing functions that rely only on the input of the real-time camera feed itself. For example, the ISP block 52 may be configured to perform automatic exposure, tone mapping, automatic focus, color correction, gamma correction, shading correction, noise reduction, black level adjustment, demosaicing, image sharpening, high dynamic range (HDR) correction, color space conversion and / or other image signal processing functions for controlling the exposure setting of the pass-through feed to output a corresponding processed pass-through feed (e.g., a series of processed video frames). The processed pass-through feed (sometimes referred to as pass-through content) may be presented to the user as a real-time video stream via one or more displays 14.

[0042] The tracking sensor 54 may include a gaze tracking subsystem, sometimes referred to as a gaze tracker, which is configured to collect gaze information or gaze point information. The gaze tracker may employ one or more inwardly facing cameras and / or other gaze tracking components (e.g., components facing the eyes and / or other light sources that emit light beams so that reflections of light beams from the user's eyes can be detected) to monitor the user's eyes. One or more gaze tracking sensors 54 may face the user's eyes and may track the user's gaze. The camera in the gaze tracking subsystem 54 may determine the position of the user's eyes (e.g., the center of the user's pupil), may determine the direction in which the user's eyes are oriented (the direction of the user's gaze), may determine the user's pupil size (e.g., so that light modulation and / or other optical parameters are adjusted based on pupil size, and / or the sequential amount used to spatially adjust one or more of these parameters, and / or the area in which one or more of these optical parameters are adjusted), may be used to monitor the current focal length of the lens in the user's eyes (e.g., whether the user is focusing on the near field or the far field, which can be used to assess whether the user is daydreaming or thinking strategically or tactically), and / or other gaze information. The camera in gaze tracker 54 may sometimes be referred to as an inward-facing camera, a gaze detection camera, an eye tracking camera, a gaze tracking camera, or an eye monitoring camera. If desired, other types of image sensors (e.g., infrared and / or visible light emitting diodes and photodetectors, etc.) may also be used to monitor the user's gaze.

[0043] The tracking sensors 54 may also include a face and body tracking subsystem that is configured to perform face tracking (e.g., to capture images of the user's jaw, mouth, etc. when the device is worn on the user's head) and body tracking (e.g., by capturing images of the user's torso, arms, hands, legs, etc. when the device is worn on the user's head). If desired, the face and body tracking subsystem may also track the user's head pose by directly determining any movement, yaw, pitch, roll, etc., of the head mounted device 10. The yaw, roll, and pitch of the user's head may collectively define the user's head pose.

[0044] Tracking sensors 54 may also include a hand tracking subsystem, sometimes referred to as a hand tracker, that is configured to monitor the user's hand movements / gestures to obtain hand gesture data. For example, a hand tracker may include a camera and / or other gesture tracking components (e.g., an outward-facing component and / or a light source that emits a light beam so that reflections of the light beam from the user's hand can be detected) to monitor the user's hand. One or more hand tracking sensors 54 may be directed toward the user's hand and may track motions associated with the user's hand, may determine whether the user is performing a tapping or swiping motion with his / her fingertips or hand, may determine whether the user is performing a non-contact button press or object selection operation with his / her hand, may determine whether the user is performing a grasping or gripping motion with his / her hand, may determine whether the user is using his / her hand or fingers to point to or pinch a given object presented on display 14, may determine whether the user is performing a swiping or banging motion with his / her hand, or may generally measure / monitor three-dimensional non-contact gestures ("air gestures") associated with the user's hand. Tracking sensors 54 operable to obtain gaze, posture, hand gestures, and other information related to the movements of a user of device 10 are sometimes collectively referred to as “user tracking” sensors.

[0045] Data collected using tracking sensors 54 may optionally be processed using a neural processing unit (NPU) 56, which is optimized for neural network processing operations. Neural processing unit 56 may perform neural network processing tasks much faster than CPU 60. Neural processing unit 56 may employ one or more machine learning-based models to output corresponding gaze information, head posture information, hand information, and / or other information related to the user.

[0046] Figure 3 The example in which the front camera 50 (e.g., an image sensor for obtaining pass-through content) and the tracking sensor 54 (e.g., an optical sensor for obtaining gaze, gesture, and / or other user-related data) are shown as separate independent subsystems is illustrative. In some embodiments, one or more cameras 50 for obtaining a real-time video pass-through feed may also be used to obtain gesture information, hand information, and / or other user-related data.

[0047] The gaze information and / or other user-related information output from the neural processing unit 56 may be transmitted to a central processing unit (CPU) 60. During normal operation, most software applications running on the device 10 employ the CPU 60 in one way or another. The CPU 60 is sometimes referred to as an application processor. Unlike the NPU 56, which is optimized for artificial neural network calculations, and the GPU 70, which is optimized for graphics rendering calculations, the CPU 60 may be configured to handle general-purpose tasks, including but not limited to running the main operating system (OS) of the device 10, executing web browsing applications, social media applications, messaging applications, media streaming applications, and mapping and navigation applications, as well as supporting wireless communications, security and encryption functions, and power management functions (to name a few).

[0048] exist Figure 3 In an example of , a fovea subsystem such as a dynamic fovea block 62 may be executed on the CPU 60. The fovea block 62 is a component configured to optimize the display rendering process by allocating more computing resources to the display area aligned with the user's gaze point, while reducing details in the peripheral area that is not aligned with the user's gaze point (e.g., by locally enhancing the image resolution of the video feed only in the area of ​​the user's gaze). Since the area or point of gaze may change over time, the fovea may be dynamically executed at a rate sufficient to keep up with the drift of the user's gaze. The fovea block 62 may, for example, be configured to generate a fovea curve and a desired display pixel grouping based on the received gaze information for locally enhancing the image resolution of the video feed in the user's gaze area.

[0049] The CPU 60 may also include a software or hardware subsystem such as a block 64, which is configured to generate one or more warp grids based on the received gaze information and / or other user-related information. Therefore, block 64 is sometimes referred to as a warp grid generation block or a warp grid generator. The warp grid generated by block 64 may refer to and is defined herein as a grid or mesh of interconnected points (vertices) that can be overlaid on top of an image. Each vertex in the warp grid can be independently adjusted to apply distortion, deformation, distortion, deformation, ripples and / or other transformations or visual effects to the underlying image. The density of vertices in the warp grid may vary depending on the level of detail currently adopted by the specific transformation. Each vertex in the warp grid may have associated deformation parameters along with its position coordinates, which may include texture coordinates, blending weights and / or other values ​​that may affect how each vertex contributes to the final warped image. In the context of the head-mounted device 10, the warp grid output by block 64 can be used for lens distortion correction, point of view correction (POVC) and / or other adjustments to the displayed pass-through feed to create a more natural, pleasant, comfortable and immersive viewing experience. Generally, a warp grid may be generated based on gaze data (e.g., the warp grid may change based on the user's gaze, head pose, and / or other movements). The warp grid may also be adjusted based on the foveal curve so that certain areas of the image are rendered with lower detail to save computational resources.

[0050] Unlike the CPU 60 and the NPU 56, the graphics processing unit (GPU) 70 is optimized for graphics processing or graphics rendering operations, as its name implies. The graphics processing unit 70 may include highly parallelized processing cores configured to handle a wide range of computationally intensive tasks. A graphics rendering subsystem such as a graphics rendering engine 72 may be implemented on the GPU 70. The graphics rendering engine 72, sometimes referred to as a graphics renderer or graphics rendering pipeline, may be configured to render or generate virtual content (e.g., virtual reality content, augmented reality content, mixed reality content, or extended reality content) or may be used to perform other graphics processing functions. The virtual (computer-generated) content output from the graphics rendering engine 72 may be optionally foveated based on the foveal information output from the dynamic foveal block 62 (e.g., the subsystem 72 may render foveated virtual content).

[0051] The graphics renderer 72 may synthesize a photorealistic or non-photorealistic image from one or more 2- or 3-dimensional models defined in a scene file that contains information about how to simulate various features, such as shading (e.g., how the color and brightness of a surface varies with lighting), shadowing (e.g., how shadows are cast on objects), texture mapping (e.g., how details are applied to surfaces), reflections, transparency or opacity (e.g., how light is transmitted through solid objects), translucency (e.g., how light is scattered through solid objects), refraction and diffraction, depth of field (e.g., how certain objects may appear out of focus when outside the depth of view), motion blur (e.g., how certain objects may appear blurry due to rapid motion), and / or other visible features related to lighting or physical properties of objects in the scene. The graphics renderer 72 may apply a rendering algorithm, such as rasterization, ray casting, ray tracing, radiosonde, or other graphics processing algorithms.

[0052] exist Figure 3 In an embodiment of the present invention, the graphics rendering block 72 may have associated rendering parameters sometimes referred to and defined herein as a "virtual camera". The virtual camera 74 associated with the graphics rendering pipeline may represent constraints imposed on the graphics rendering operation, which define the perspective or viewpoint used to render virtual content for the user's eyes. For example, the virtual camera 74 may include: (1) a first virtual (rendering) camera 74-1, which determines the perspective of virtual content generated from it to be rendered for the left eye (sometimes referred to and defined herein as "left eye content" or left eye virtual content), and (2) a second virtual (rendering) camera 74-2, which determines the perspective of virtual content generated from it to be rendered for the right eye (sometimes referred to and defined herein as "right eye content" or right eye virtual content). Therefore, the left virtual camera 74-1 is sometimes referred to as the first point of view (POV) or perspective rendering parameters in this article, and the right virtual camera 74-2 is sometimes referred to as the second POV or perspective rendering parameters in this article. The virtual rendering cameras are therefore sometimes collectively referred to as view rendering parameters or constraints.

[0053] Image warping operations may be performed at content warping subsystem 80, sometimes referred to as a warping block. Subsystem 80 may represent a software or hardware component configured to receive the warping grid output from warping grid generation block 64 via path 66 and to warp the pass-through content and virtual content (e.g., applying distortion, deformation, warping, distorting, ripples, and / or other transformations or visual effects to the underlying image). Figure 3In an example of , the warp block 80 may include a first subcomponent 82 configured to warp pass-through content and a second subcomponent 84 configured to warp virtual content separately. Specifically, the first warp subcomponent 82 may receive the pass-through content from the output of the ISP 52, and warp the pass-through content based on the received warp grid and / or other information related to the warping of the real-time pass-through feed to produce corresponding warped pass-through content. The second warp subcomponent 84 may receive the rendered virtual content from the output of the GPU 70, and warp the virtual content based on the received warp grid and / or other information related to the warping of the computer-generated content to produce corresponding warped virtual content.

[0054] Media merging compositor 90 may receive the warped pass-through content and the warped virtual content from warping subsystem 80, and may overlay, blend, or otherwise combine one or more portions of the warped virtual content with the warped pass-through content to obtain corresponding merged or blended video content. The merged video content output from media merging compositor 90 may then be presented on display 14 for viewing by a user of device 10.

[0055] Combination Figures 1 to 3 Devices 10 of the type described may be configured to provide diplopia correction (sometimes referred to as diplopia compensation or diplopia mitigation). The diplopia phenomenon is typically due to misalignment of the user's eyes, such as when the left eye and the right eye are not pointing in the same direction. According to one embodiment, diplopia may be mitigated by shifting the projected content of one of the virtual cameras. This approach is useful in Figure 4 The top (bird's eye view) view of Figure 4 As shown, the first virtual camera 100-1 may have a first viewing cone represented by line 102, and the second virtual camera 100-2 may have a second viewing cone represented by line 104. The virtual cameras 100-1 and 100-2 may represent a combination of Figure 3 A left virtual (rendering) camera 74-1 and a right virtual (rendering) camera 74-2 are depicted. A "view frustum" may refer to and be defined herein as a geometric shape that represents the volume of space that a particular camera can capture in a three-dimensional (3D) scene (e.g., the view frustum may define the field of view of each virtual camera). The view frustum may have a truncated pyramid shape defined by six planes: a near plane, a far plane, a top plane joining the upper edges of the near plane and the far plane, a bottom plane joining the lower edges of the near plane and the far plane, a left plane joining the left edges of the near plane and the far plane, and a right plane joining the right edges of the near plane and the far plane. The near plane of the view frustum may be aligned with Figure 4 The projection plane 110 shown in the top view of FIG. The projection plane 110 may therefore represent the plane within the viewing frustum that is closest to the camera and is the plane on which the 3D scene is projected to create the final two-dimensional (2D) image.

[0056] exist Figure 4 In the example of , the first view frustum 102 of virtual camera 100-1 and the second view frustum 104 of virtual camera 100-2 may coincide with each other, as shown by the alignment of the two view frustums along the projection plane 110. However, this perfect alignment of the left and right view frustums may not correspond to an actual misalignment between the eyes of a user having diplopia or other medical condition that may cause a user to experience diplopia. To help mimic such a misalignment or divergence between the eyes of a user, the projected content of one of the virtual cameras may be shifted as perceived by one of the user's eyes (e.g., to shift the position of the rendered virtual content within the user's field of view on a display). This lateral shifting of the rendered content may be achieved by modifying the view frustum limits (e.g., by modifying the angular limits or boundaries of the view frustum without changing the position of the virtual cameras). This type of shifting is not particularly useful in simulating a user's eyes. Figure 4 1, the frustum of the second virtual camera 100-2 is now offset relative to the frustum of the first virtual camera 100-1, as shown in FIG. 1 (as indicated by arrow 108), such that the frustum of the second virtual camera 100-2, now indicated by dashed line 106, is offset relative to the frustum of the first virtual camera 100-1. This perceived content shift achieved by adjusting the frustum limit angle is exemplary. Alternatively, the frustum of one of the virtual cameras may be shifted laterally (e.g., by laterally shifting the position of one of the virtual cameras) to shift the projected content on the display.

[0057] Figure 4 The example in which the projected / rendered content of virtual camera 100-2 is shifted laterally to the right along projection plane 110 is merely illustrative. In general, the rendered content of virtual camera 100-1 may be shifted laterally in any direction along projection plane 110 and / or the rendered content of virtual camera 100-2 may be shifted laterally in any direction along projection plane 110. If desired, the projected content of both virtual cameras 100-1 and 100-2 may be optionally shifted. This technique in which the content of one of the virtual cameras is shifted without deviating from the projection (near) plane is sometimes referred to as planar projection shifting. Generating virtual content by shifting or otherwise modifying the content of one or more of the virtual rendering cameras may help alleviate double vision for a user viewing the virtual content.

[0058] Can be used Figure 4An example of shifting the projected content of a virtual camera to correct double vision associated with virtual content generated by a graphics rendering pipeline. A similar shift in the field of view may also be employed to mitigate double vision for real-world pass-through content. For example, the pass-through content may be shifted laterally by selectively cropping the left and right images at the image signal processor 52. The pass-through content may be collectively referred to as a series of pass-through images presented to the left eye (sometimes referred to and defined herein as "left eye content" or left eye pass-through content) and a series of pass-through images presented to the right eye (sometimes referred to and defined herein as "right eye content" or right eye pass-through content). Multiple separate front-facing cameras 50 may be used (see Figure 3 ) or using one front camera 50 to capture these pass-through images intended for the left and right eyes. The left pass-through image and / or the right pass-through image may be cropped individually to achieve the desired planar projection (or field of view) shift. This assumes that the pre-cropped image has enough field of view to accommodate the cropping amount used to align the field of view of the left pass-through image with the field of view of the right pass-through image.

[0059] Figure 4 The examples related to shifting the projected content along the projection plane or shifting the field of view to perform diplopia compensation are illustrative. Figure 5 is a bird's-eye view showing another embodiment in which double vision can be alleviated by transforming the point of view (POV) of one or more of the virtual cameras. The terms "point of view" (POV), "angle of view", and "pose" related to cameras are sometimes used interchangeably herein. Figure 5 As shown, the first virtual camera 100-1 may have a first viewing cone represented by line 102, and the second virtual camera 100-2 may have a second viewing cone represented by dashed line 122. The virtual cameras 100-1 and 100-2 may represent a combination of Figure 3 Depicted are a left virtual (rendering) camera 74 - 1 and a right virtual (rendering) camera 74 - 2 .

[0060] The second view frustum 122 may be transformed relative to the first view frustum 102, as indicated by the direction of arrow 120, and is therefore sometimes referred to as a transformed view frustum. The first view frustum 102 may have a first projection plane 110, while the second view frustum 122 may have a different projection plane 110' that is transformed relative to the projection plane 110. This orientation difference between the two virtual cameras may be referred to as a difference in camera pose, which includes a 3D rotation and / or translation relative to a common coordinate system. The term "transformation" may be defined herein to include a 3D rotation and / or translation. This transformation of the view frustum may help mimic such a misalignment or divergence between the user's eyes so that the user will no longer experience double vision. In Figure 5In the example of , the view cone of virtual camera 100-2 may be rotated to the right about the Y axis (as indicated by arrow 120). This is merely illustrative. In general, the pose of virtual camera 100-1 or 100-2 may be rotated left or right about the Y axis, rotated up or down about the X axis, and / or tilted about the Z axis. Additionally or alternatively, the position of virtual camera 100-1 or 100-2 may be translated in any direction along the X axis, Y axis, or Z axis. If desired, the pose or viewing angle of both virtual cameras 100-1 and 100-2 may be optionally rotated. Generating virtual content by rotating the POV of one or more of the virtual rendering cameras may help alleviate double vision for a user viewing the virtual content.

[0061] Can be used Figure 5 Examples of transformations related to virtual cameras are used to correct for double vision associated with virtual content generated by the graphics rendering pipeline. Similar transformations in camera perspective can also be used to mitigate double vision for real-world pass-through content. For example, warping or depth reprojection techniques (e.g., using Figure 3 82 in ) to transform the pass-through content to generate views of the scene from different viewpoints. As another example (as an alternative to the software-only solution described above), the device 10 may be provided with adaptive hardware, such as a camera actuator 150, for physically rotating the left camera 100-1 and the right camera 100-2 (assuming that cameras 100-1 and 100-2 now represent front cameras for capturing pass-through content). For example, the actuator 150 may be configured to allow two (left and right) separate optical modules to rotate around a center located between the lens and the sensor of each optical module. The actuator 150 rotates each front camera around the X-axis, around the Y-axis, and / or around the Z-axis. This example of physically rotating the front camera using the actuator 150 is illustrative. In another embodiment, the device 10 may be provided with a device configured to physically rotate the entire optical module 40 (see, e.g. Figure 1 ). For example, the actuator may be used to transform (e.g., rotate and / or translate) an optical module 40 that may include a display 14, a lens 30, and other support structures in an optical stack for one or both eyes. Such rotation may be performed along a curved 2D surface (e.g., the surface of a segment of a sphere) in accordance with a forward gaze rotation of the eye (e.g., about the center of the eyeball).

[0062] The embodiments described herein can be used for different types of displays 14. For example, one or more displays 14 in device 10 can be flat (planar) displays. As another example, one or more displays 14 in device 10 can be curved displays. Such curved displays may include spherical displays (e.g., displays having a surface similar to a segment of a sphere or hemisphere). Such curved displays may optionally be shifted or transformed for further optimization. Content may be presented only on a dedicated area of ​​such a curved display based on the forward gaze of the eye (e.g., the field of view may be dynamically adjusted on the curved display). If desired, other components in the optical module (including the lens and any eye tracking sensors) may also be rotated or moved based on the forward gaze of the eye to utilize the entire visible area of ​​the curved display through the lens. In other embodiments, sensor 16 in device 10 (see Figure 2 ) may optionally include a scene camera and / or other sensors configured to capture a full light field or other information that can be used to improve modification of the pass-through content.

[0063] Figure 6 It is used to operate the combination Figures 1 to 5 Flowchart of illustrative steps of an electronic device 10 of the type described. During the operation of block 200, one or more image sensors 50 may acquire pass-through content. For example, one or more front-facing cameras in device 10 may be configured to capture a real-time pass-through video feed of the physical environment or scene surrounding device 10.

[0064] During the operation of block 202, the image signal processor 52 may optionally be shifted (e.g., as Figure 4 ) or transformed (e.g., as shown in Figure 5) around one or more rotation axes (as shown in ). In one embodiment, the image signal processor 52 may perform a lateral shift or planar translation of the field of view (FOV) of the transparent content by selectively cropping the left and / or right transparent images. In another embodiment, the image signal processor 52 may perform a desired rotation of the transparent content by performing point of view correction (POVC). For example, the image signal processor 52 may generate a 3D model of the surrounding physical environment around the device 10. Based on this 3D model of the physical environment, the image signal processor 52 may perform a depth reprojection operation to generate a new view of the current scene from a different viewpoint or perspective. As another example, the image signal processor 52 may be configured to perform novel view synthesis based on a trained machine learning (ML) model. Given an image from a particular viewpoint of a front camera, such a machine learning model may be used to generate images of the same scene from different viewpoints. While model-based synthesis of new views may require synthesizing (as opposed to regenerating) scene content in areas occluded from the view of the front-facing camera, such artifacts may be tolerable when weighed against the disadvantage of not being able to use device 10 due to double vision.

[0065] The shifting or transformation of the pass-through content need not be performed at the image signal processor 52. If desired, the shifting or transformation of the pass-through content may alternatively be performed at the warping subsystem 80 (see block 204). During the operation of block 204, the pass-through content may optionally be shifted at the warping subsystem 80 (e.g., as shown in FIG. 204). Figure 4 ) or transformed (e.g., as shown in Figure 5 , about one or more rotation axes as shown in ). For example, the warp block 80 may be configured to selectively warp the left and / or right pass-through images based on the warp grid generated by block 64 to apply a desired warp, rotation, lateral translation, POV adjustment, distortion correction, deformation, or other modification to help compensate for double vision. Only one of blocks 202 and 204 need be executed (e.g., if block 202 is executed, block 204 may be omitted; if block 204 is executed, block 202 may be omitted). In yet another embodiment, the modification to the pass-through content may be performed (e.g., using Figure 5 This is accomplished by physically shifting, rotating, or otherwise adjusting the front-facing camera by a desired amount using an adaptive actuator 150 (shown in FIG. 1 ).

[0066] During the operation of block 206, the Figure 4 ) or rotated (e.g., as shown in Figure 5) around one or more rotation axes as shown in . For example, the view cone limit angle of at least one of the left virtual rendering camera and the right virtual rendering camera may be modified to alleviate diplopia or improve vergence comfort. As another example, the view cone of at least one of the left virtual rendering camera and the right virtual rendering camera may be shifted laterally along the projection plane to compensate for diplopia or improve vergence comfort. As another example, the posture of at least one of the left virtual rendering camera and the right virtual rendering camera may be rotated around the X-axis and / or the Y-axis to compensate for diplopia or improve vergence comfort. Any adjustment to the virtual rendering camera (sometimes referred to herein as "view rendering parameters") may change the position of the projected content on the display, the view cone, or the viewing angle from which the virtual content will be rendered.

[0067] Moving or transforming virtual content need not be performed at the graphics rendering engine 72. If desired, the shifting or transforming of virtual content may alternatively be performed at the warp subsystem 80 (see block 208). During the operation of block 208, the virtual content may optionally be shifted at the warp subsystem 80 (e.g., as shown in FIG. 208 ). Figure 4 ) or transformed (e.g., as shown in Figure 5 8 (about one or more rotation axes as shown in ). For example, the warp block 80 may be configured to selectively warp the left and / or right virtual images based on the warp grid generated by the block 64 to apply a desired warp, rotation, lateral translation, POV adjustment, distortion correction, deformation, or other modification to help compensate for diplopia or improve vergence comfort. Only one of the blocks 206 and 208 need to be executed (e.g., if the block 206 is executed, the block 208 may be omitted; if the block 208 is executed, the block 206 may be omitted).

[0068] Figure 6 The examples in which blocks 206 and 208 are shown as occurring after blocks 202 and 204 are merely illustrative. In general, the operations of blocks 206 or 208 may occur in parallel (simultaneously) with the operations of blocks 200, 202, or 204 or before them (e.g., the generation and adjustment of pass-through content and virtual content may occur in any order).

[0069] During the operation of block 210, the adjusted pass-through content and the adjusted virtual content may be mixed, merged, or otherwise combined at the media merge compositor 90. The adjusted pass-through content may represent the shifted or transformed pass-through content output from block 202 or block 204. The adjusted virtual content may represent the shifted or transformed pass-through content output from block 206 or block 208. If a plane frustum shift (or cropping) is used to modify the pass-through content, a corresponding amount of plane frustum shift should be used to modify the virtual content. Alternatively, if a perspective / POV transformation is used to modify the pass-through content, a corresponding amount of transformation should be used to modify the virtual content. The media merge compositor 90 may output the merged content (e.g., by overlaying the modified virtual content on the modified pass-through content), which is then presented on one or more displays 14 in the device 10. Displaying augmented reality content in this manner may be technically advantageous and helpful in compensating for or mitigating potential diplopia artifacts seen by users with diplopia and / or improving comfort for users experiencing vergence discomfort.

[0070] Figure 6 The operations described are merely illustrative. In some embodiments, one or more of the operations described may be modified, replaced, or omitted. In some embodiments, one or more of the operations described may be performed in parallel. In some embodiments, additional processes may be added or inserted between the operations described. If desired, the order of certain operations may be reversed or changed, and / or the timing of the operations described may be adjusted so that they occur at slightly different times. In some embodiments, the operations described may be distributed in a larger system.

[0071] The amount of shift or transformation applied to the pass-through content and the virtual content may be based on one or more diplopia compensation parameters. Figure 7 220 is a flowchart of illustrative steps for generating diplopia compensation parameters according to eyeglass prescription values ​​according to some embodiments. During the operation of block 220, device 10 may obtain prescription values ​​from a user. Eyeglass prescription values ​​associated with prism correction (sometimes referred to as "prism prescription values" or "prism correction values") may be given in units of prism diopters. In some embodiments, prism prescription values / correction values ​​may be embedded in lightweight application codes, quick response (QR) codes, or other scannable codes that users receive from optometrists along with their prescription lenses. Device 10 may scan such codes and automatically read prism prescription values ​​without manual input from the user, which may help minimize human errors and facilitate use. For example, a remote server may maintain a database linking prescription lenses to corresponding prism values, and device 10 may communicate with the remote server to obtain prism values ​​for prescription lenses that do not originally provide prism correction.

[0072] To help protect the user's privacy, best practices may be used to handle any personal user information collected by the device 10. These best practices include meeting or exceeding any applicable privacy regulations. Opt-in and opt-out options and / or other options may be provided that allow users to control the use of their personal data.

[0073] During the operation of block 222, the prism prescription value may then be converted into a corresponding compensation parameter α X and α Y As an example, the prism prescription value (in prism diopters) may be proportional to the tangent of the compensation parameter. X represents the rotation around the X axis (see e.g. Figure 5 ), which indicates the amount of posture rotation upward or downward and is therefore sometimes referred to as the vertical rotation angle. Y represents the rotation angle around the Y axis, which indicates the amount of posture rotation to the left or right and is therefore sometimes referred to as the horizontal rotation angle. If necessary, an additional compensation parameter α may be optionally obtained Z , which represents the attitude rotation around the Z axis.

[0074] During operation of block 224, device 10 may apply a necessary amount of plane shift or POV transformation to the pass-through content and virtual content based on the compensation parameters calculated from block 222. X , α Y etc.) can be used to determine the amount of lateral shift or cropping required to modify the virtual content and the transparent content, or can be used to determine the amount of transformation required to modify the virtual content and the transparent content to reduce double vision. The mechanism for performing the desired shift and transformation is combined with Figure 6 description, and need not be repeated in detail to avoid obscuring the present embodiments.

[0075] During the operation of block 226, device 10 may optionally allow the user to fine-tune the compensation parameters to minimize or further reduce any remaining diplopia artifacts. Here, the user may be provided with an opportunity to manually adjust the compensation parameters or prism prescription values ​​until he / she is satisfied (e.g., until the user's eyes are able to fuse the split vision). For example, the user may be presented with a step size comparable to that used by an optometrist (e.g., in steps of 0.5 prism diopters, less than 0.5 prism diopters, 0.5 to 1 prism diopters, less than 1 prism diopters, less than 2 prism diopters, less than 5 prism diopters, etc.) to adjust the vertical compensation parameter α. X and / or horizontal compensation parameter α Y The user can, for example, use a keyboard, mouse, touch pad and / or other input-output device 24 (see Figure 2) to provide such external input. Generally speaking, the compensation parameters can be adjusted or determined based on input or feedback from the user.

[0076] Figure 7 The examples in which diplopia compensation parameters are calculated based on provided prism prescription values ​​are illustrative. Figure 8 Another embodiment for generating diplopia compensation parameters based on measured gaze data is illustrated. During the operation of block 250, device 10 can display stimulation points for user focus. During the operation of block 252, device 10 can track or monitor the user's gaze point to estimate any misalignment or deviation from the expected gaze point (the expected gaze point should be a function of the position of the stimulation point). Optionally, the user can provide some indication about him / her focusing, and data can be captured at this point in time. Multiple measurements or data sampling (sometimes referred to as "gaze data points") can be performed in this way. During the operation of block 254, outliers in the collected data samples can be optionally removed or discarded. Then the remaining data samples after removing the abnormal data points can be optionally averaged to reduce measurement errors.

[0077] During the operation of block 256, the estimated misalignment data output from the operations of blocks 252 / 254 may then be converted into corresponding compensation parameters α X and α Y The compensation parameter α X represents the rotation around the X axis (see e.g. Figure 5 ), which indicates the amount of posture rotation upward or downward and is therefore sometimes referred to as the vertical rotation angle. Y represents the rotation angle around the Y axis, which indicates the amount of posture rotation to the left or right and is therefore sometimes referred to as the horizontal rotation angle. If necessary, an additional compensation parameter α may be optionally obtained Z , which represents the amount of posture rotation about the Z axis. The operations of blocks 250, 252, 254, and 256 may be repeated for the left eye and separately for the right eye.

[0078] During operations at block 258, device 10 may apply a necessary amount of plane shift or perspective transformation to the pass-through content and the virtual content based on the compensation parameters calculated from block 256. X , α Y etc.) can be used to determine the amount of lateral shift or cropping required to modify the virtual content and the transparent content, or can be used to determine the amount of transformation required to modify the virtual content and the transparent content to reduce double vision. The mechanism for performing the desired shift and transformation is combined with Figure 6 description, and need not be repeated in detail to avoid obscuring the present embodiments.

[0079] During the operation of block 260, device 10 may optionally allow the user to fine-tune the compensation parameters to minimize or further reduce any remaining diplopia artifacts. Here, the user may be provided with an opportunity to manually adjust the compensation parameters until he / she is satisfied (e.g., until the user's eyes are able to fuse the split vision). For example, the user may be presented with options for adjusting the vertical compensation parameter α in steps comparable to those used by optometrists (e.g., in steps of 0.5 prism diopters, less than 0.5 prism diopters, 0.5 to 1 prism diopters, less than 1 prism diopters, less than 2 prism diopters, less than 5 prism diopters, etc.). X and / or horizontal compensation parameter α Y The user can, for example, use a keyboard, mouse, touch pad and / or other input-output device 24 (see Figure 2 ) to provide such external input. Generally speaking, the compensation parameters can be adjusted or determined based on input or feedback from the user.

[0080] Combination of the above Figures 1 to 8 The described methods and operations may be performed by the components of device 10 using software, firmware, and / or hardware (e.g., dedicated circuits or hardware). The software code for performing these operations may be stored on a non-transitory computer-readable storage medium (e.g., a tangible computer-readable storage medium) stored on one or more of the components of device 10 (e.g., a computer-readable storage medium). Figure 1 The software code may sometimes be referred to as software, data, instructions, program instructions, or code. Non-transitory computer-readable storage media may include a drive, non-volatile memory such as non-volatile random access memory (NVRAM), a removable flash drive or other removable media, other types of random access memory, etc. The software stored on the non-transitory computer-readable storage medium may be executed by processing circuitry on one or more of the components of the device 10 (e.g., one or more processors in the control circuitry 20). The processing circuitry may include a microprocessor, an application processor, a digital signal processor, a central processing unit (CPU), an application-specific integrated circuit having a processing circuit, or other processing circuitry.

[0081] There are many different types of electronic systems that can enable a person to sense and / or interact with various XR environments. Examples include head-mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers.

[0082] According to one embodiment, a method for operating an electronic device is provided, the method comprising: acquiring transparent content using one or more image sensors; generating virtual content using a graphics rendering subsystem; reducing diplopia or improving vergence comfort by at least modifying the transparent content to produce corresponding modified transparent content; merging the modified transparent content with the virtual content to produce corresponding merged content; and presenting the merged content using one or more displays.

[0083] According to another embodiment, the pass-through content optionally includes left eye content and right eye content, and modifying the pass-through content includes cropping a selected one of the left eye content and the right eye content at an image signal processor to reduce diplopia or improve vergence comfort.

[0084] According to another embodiment, the pass-through content optionally includes left eye content and right eye content, and modifying the pass-through content includes transforming a selected one of the left eye content and the right eye content at an image signal processor to reduce diplopia or improve convergence comfort.

[0085] According to another embodiment, the pass-through content optionally includes left-eye content and right-eye content, and modifying the pass-through content includes selectively distorting at least one of the left-eye content and the right-eye content to apply a lateral shift in the field of view of at least one of the left-eye content and the right-eye content.

[0086] According to another embodiment, the pass-through content optionally includes left-eye content and right-eye content, and modifying the pass-through content includes selectively distorting at least one of the left-eye content and the right-eye content to apply a transformation in the viewpoint of the at least one of the left-eye content and the right-eye content.

[0087] According to another embodiment, the virtual content optionally includes left-eye content and right-eye content, and the graphics rendering subsystem is optionally configured to generate the virtual content based on a first virtual camera and a second virtual camera, wherein the first virtual camera imposes constraints limiting a first viewpoint from which the left-eye content is rendered at the graphics rendering subsystem, and the second virtual camera imposes constraints limiting a second viewpoint from which the right-eye content is rendered at the graphics rendering subsystem.

[0088] According to another embodiment, the first virtual camera optionally has a first viewing cone and the second virtual camera optionally has a second viewing cone, and the method optionally includes: utilizing the graphics rendering subsystem to shift a selected one of the left eye content and the right eye content on the one or more displays by adjusting the angle limits of corresponding viewing cones in the first viewing cone and the second viewing cone to reduce diplopia or improve convergence comfort.

[0089] According to another embodiment, the first virtual camera optionally has a first viewing cone and the second virtual camera optionally has a second viewing cone, and the method optionally includes: utilizing the graphics rendering subsystem to laterally shift a selected one of the first viewing cone and the second viewing cone along a common projection plane to reduce diplopia or improve convergence comfort.

[0090] According to another embodiment, the method optionally includes: utilizing the graphics rendering subsystem to transform a selected one of the first virtual camera and the second virtual camera to alleviate diplopia or improve vergence comfort.

[0091] According to another embodiment, the virtual content optionally includes left eye content and right eye content, and the method optionally includes: modifying the virtual content by selectively distorting at least one of the left eye content and the right eye content to apply a lateral shift in the field of view of at least one of the left eye content and the right eye content.

[0092] According to another embodiment, the virtual content optionally includes left-eye content and right-eye content, and the method optionally includes: optionally modifying the virtual content by selectively distorting at least one of the left-eye content and the right-eye content to apply a transformation in the viewpoint of at least one of the left-eye content and the right-eye content.

[0093] According to another embodiment, the method optionally includes: obtaining gaze data using one or more tracking sensors; generating a distorted grid based on the gaze data, wherein modifying the transparent content includes modifying the transparent content based on the distorted grid to reduce diplopia or improve vergence comfort; and modifying the virtual content based on the distorted grid to reduce diplopia or improve vergence comfort.

[0094] According to one embodiment, a method for operating an electronic device is provided, the method comprising: acquiring transparent content using one or more image sensors; generating virtual content using a graphics rendering subsystem; obtaining prism prescription values ​​for correcting diplopia or improving convergence comfort; calculating compensation parameters based on the prism prescription values; modifying the transparent content based on the compensation parameters; and displaying a mixed version of the modified transparent content and the virtual content.

[0095] According to another embodiment, the pass-through content optionally includes left-eye content and right-eye content, and modifying the pass-through content optionally includes shifting or transforming one or more of the left-eye content and the right-eye content to reduce diplopia or improve vergence comfort.

[0096] According to another embodiment, the method optionally includes: modifying the virtual content based on the compensation parameters, wherein the virtual content includes left eye content and right eye content, and wherein modifying the virtual content includes shifting or transforming one or more of the left eye content and the right eye content to reduce diplopia or improve convergence comfort.

[0097] According to another embodiment, the method optionally includes providing the user with an opportunity to adjust compensation parameters to minimize diplopia or improve vergence comfort.

[0098] According to another embodiment, the method optionally includes: reading a scannable code; and obtaining the prism prescription value from the scannable code.

[0099] According to one embodiment, a method for operating an electronic device is provided, the method comprising: acquiring transparent content using one or more image sensors; utilizing a graphics rendering subsystem; generating virtual content; displaying stimulation points; while the stimulation points are being displayed, monitoring gaze points to estimate misalignment between the gaze points and expected gaze points; calculating compensation parameters based on the estimated misalignment; modifying the transparent content based on the compensation parameters; and displaying a mixed version of the modified transparent content and the virtual content.

[0100] According to another embodiment, the pass-through content optionally includes left-eye content and right-eye content, and modifying the pass-through content optionally includes shifting or transforming one or more of the left-eye content and the right-eye content to reduce diplopia or improve vergence comfort.

[0101] According to another embodiment, the method optionally includes: modifying the virtual content based on the compensation parameters, wherein the virtual content includes left eye content and right eye content, and wherein modifying the virtual content includes shifting or transforming one or more of the left eye content and the right eye content to reduce diplopia or improve convergence comfort.

[0102] According to another embodiment, the method optionally includes providing the user with an opportunity to adjust compensation parameters to minimize diplopia.

[0103] According to another embodiment, the method includes estimating the misalignment by obtaining a plurality of fixation data points, removing one or more outliers from the plurality of fixation data points, and averaging the plurality of fixation data points.

[0104] According to an embodiment, a method for operating an electronic device is provided, the method comprising: generating virtual content having left-eye content and right-eye content using a graphics rendering subsystem, wherein the graphics rendering subsystem is configured to generate the virtual content based on a first virtual camera and a second virtual camera, the first virtual camera applying a constraint limiting a first viewpoint from which the left-eye content is rendered at the graphics rendering subsystem, and the second virtual camera applying a constraint limiting a second viewpoint from which the right-eye content is rendered at the graphics rendering subsystem; reducing diplopia or improving vergence comfort by modifying a selected one of the left-eye content and the right-eye content to produce the modified virtual content; and outputting the modified content using one or more displays.

[0105] According to another embodiment, modifying a selected one of the left-eye content and the right-eye content optionally includes shifting the left-eye content or the right-eye content on the one or more displays.

[0106] According to another embodiment, modifying the selected one of the left eye content and the right eye content optionally includes transforming the first virtual camera or the second virtual camera.

[0107] According to another embodiment, modifying a selected one of the left-eye content and the right-eye content optionally includes warping the left-eye content or the right-eye content to apply a lateral shift.

[0108] According to another embodiment, modifying a selected one of the left-eye content and the right-eye content optionally includes warping the left-eye content or the right-eye content to apply a perspective transform.

[0109] The foregoing is merely illustrative and various modifications may be made to the embodiments described. The foregoing embodiments may be implemented independently or in any combination.

Claims

1. A method for operating an electronic device, the method comprising: Using one or more image sensors to obtain transparent content; Generate virtual content using a graphics rendering subsystem; Reducing diplopia or improving vergence comfort by at least modifying the transparent content to produce corresponding modified transparent content; Merging the modified pass-through content with the virtual content to generate corresponding merged content; as well as The merged content is presented using one or more displays.

2. The method according to claim 1, wherein: The transparent content includes left-eye content and right-eye content; and Modifying the pass-through content includes cropping a selected one of the left-eye content and the right-eye content at an image signal processor to reduce diplopia or improve vergence comfort.

3. The method according to claim 1, wherein: The transparent content includes left-eye content and right-eye content; and Modifying the pass-through content includes transforming a selected one of the left-eye content and the right-eye content at an image signal processor to reduce diplopia or improve vergence comfort.

4. The method according to claim 1, wherein: The transparent content includes left-eye content and right-eye content; and Modifying the pass-through content includes selectively distorting at least one of the left-eye content and the right-eye content to apply a lateral shift in a field of view of the at least one of the left-eye content and the right-eye content.

5. The method according to claim 1, wherein: The transparent content includes left-eye content and right-eye content; and Modifying the pass-through content includes selectively warping at least one of the left-eye content and the right-eye content to apply a transformation in a viewpoint of the at least one of the left-eye content and the right-eye content.

6. The method according to claim 1, wherein: The virtual content includes left-eye content and right-eye content; and The graphics rendering subsystem is configured to generate the virtual content based on a first virtual camera and a second virtual camera, wherein the first virtual camera imposes constraints defining a first viewpoint from which the left eye content is rendered at the graphics rendering subsystem, and the second virtual camera imposes constraints defining a second viewpoint from which the right eye content is rendered at the graphics rendering subsystem.

7. The method of claim 6, wherein the first virtual camera has a first viewing frustum and wherein the second virtual camera has a second viewing frustum, the method further comprising: A selected one of the left eye content and the right eye content is shifted on the one or more displays to reduce diplopia or improve vergence comfort by adjusting angular limits of corresponding ones of the first and second viewing cones using the graphics rendering subsystem.

8. The method of claim 6, wherein the first virtual camera has a first viewing frustum and wherein the second virtual camera has a second viewing frustum, the method further comprising: A selected one of the first viewing cone and the second viewing cone is displaced laterally along a common projection plane to alleviate diplopia or improve vergence comfort using the graphics rendering subsystem.

9. The method according to claim 6, further comprising: Using the graphics rendering subsystem, a selected one of the first virtual camera and the second virtual camera is transformed to alleviate diplopia or improve vergence comfort.

10. The method according to claim 6, wherein the virtual content includes left-eye content and right-eye content, and the method further comprises: The virtual content is modified by selectively warping at least one of the left-eye content and the right-eye content to apply a lateral shift in a field of view of the at least one of the left-eye content and the right-eye content.

11. The method according to claim 1, wherein the virtual content includes left-eye content and right-eye content, and the method further comprises: The virtual content is modified by selectively warping at least one of the left-eye content and the right-eye content to apply a transformation in a viewpoint of the at least one of the left-eye content and the right-eye content.

12. The method according to claim 1, further comprising: obtaining gaze data using one or more tracking sensors; generating a warped grid based on the gaze data, wherein modifying the pass-through content comprises modifying the pass-through content based on the warped grid to reduce diplopia or improve vergence comfort; as well as The virtual content is modified based on the warped grid to alleviate diplopia or improve vergence comfort.

13. A method of operating an electronic device, the method comprising: Using one or more image sensors to obtain transparent content; Generate virtual content using a graphics rendering subsystem; Obtain prism prescription values ​​for correcting diplopia or improving vergence comfort; Calculating compensation parameters based on the prism prescription values; Modify the transparent transmission content based on the compensation parameter; as well as A mixed version of the modified transparent content and the virtual content is displayed.

14. The method according to claim 13, wherein: The transparent transmission content includes left-eye content and right-eye content; Modifying the pass-through content includes shifting or transforming one or more of the left-eye content and the right-eye content to reduce diplopia or improve vergence comfort.

15. The method according to claim 13, further comprising: The virtual content is modified based on the compensation parameters, wherein the virtual content includes left eye content and right eye content, and wherein modifying the virtual content includes shifting or transforming one or more of the left eye content and the right eye content to reduce diplopia or improve vergence comfort.

16. The method according to claim 13, further comprising: The user is provided with the opportunity to adjust the compensation parameters to minimize diplopia or improve vergence comfort.

17. The method according to claim 13, further comprising: Read scannable codes; as well as The prism prescription value is obtained from the scannable code.

18. A method of operating an electronic device, the method comprising: Using one or more image sensors to obtain transparent content; Generate virtual content using a graphics rendering subsystem; Display stimulation points; monitoring a fixation point to estimate a misalignment between the fixation point and an expected fixation point while the stimulus point is being displayed; calculating compensation parameters based on the estimated misalignment; Modify the transparent transmission content based on the compensation parameter; as well as A mixed version of the modified transparent content and the virtual content is displayed.

19. The method according to claim 18, further comprising: The user is provided with the opportunity to adjust the compensation parameters to minimize diplopia.

20. A method of operating an electronic device, the method comprising: generating, with a graphics rendering subsystem, virtual content having left-eye content and right-eye content, wherein the graphics rendering subsystem is configured to generate the virtual content based on a first virtual camera and a second virtual camera, the first virtual camera imposing a constraint defining a first viewpoint from which the left-eye content is rendered at the graphics rendering subsystem, and the second virtual camera imposing a constraint defining a second viewpoint from which the right-eye content is rendered at the graphics rendering subsystem; reducing diplopia or improving vergence comfort by modifying a selected one of the left-eye content and the right-eye content to generate modified virtual content; as well as The modified content is output using one or more displays.

Images