Binocular parallax correction system

Abstract

The invention relates to a binocular parallax correction system. The disclosed computer-implemented method may include displaying a first test pattern on a first display and a second test pattern on a second display, and acquiring the displayed first test pattern and the displayed second test pattern. The method may also include determining a deviation between at least one of the acquired first test pattern and the first test pattern, or the acquired second test pattern and the second test pattern. The method may also include adjusting an output to at least one of the first display or the second display based on the determined deviation. Various other methods, systems, and computer readable media are also disclosed.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to U.S. Non - Provisional Patent Application No. 18 / 521,977, filed on November 28, 2023, the entire content of which is incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to binocular disparity detection and correction. Background Art

[0004] Artificial reality devices typically adopt a wearable form factor with near - eye optics to place a display near a user's eyes. Summary of the Invention

[0005] According to one aspect of the present disclosure, a computer - implemented method is provided, including: displaying a first test pattern on a first display and a second test pattern on a second display; acquiring the displayed first test pattern and the displayed second test pattern; determining a deviation between at least one of the following: the acquired first test pattern and the first test pattern, or the acquired second test pattern and the second test pattern; and adjusting an output to at least one of the first display or the second display based on the determined deviation.

[0006] According to another aspect of the present disclosure, a system is provided, including: at least one physical processor; a first display; a second display; an image sensor device; and a physical memory including computer - executable instructions that, when executed by the physical processor, cause the physical processor to: display a first test pattern on the first display and a second test pattern on the second display; acquire the displayed first test pattern and the displayed second test pattern using the image sensor device; determine a deviation between at least one of the following: the acquired first test pattern and the first test pattern, or the acquired second test pattern and the second test pattern; and adjust an output to at least one of the first display or the second display based on the determined deviation.

[0007] In accordance with yet another aspect of the present disclosure, there is provided a non-transitory computer-readable medium including one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to: display a first test pattern on a first display and a second test pattern on a second display; collect the displayed first test pattern and the displayed second test pattern; determine a deviation between at least one of the following: the collected first test pattern and the first test pattern, or the collected second test pattern and the second test pattern; and adjust an output to at least one of the first display or the second display based on the determined deviation. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The drawings illustrate various exemplary embodiments and are a part of the specification. These drawings, together with the following description, demonstrate and explain the various principles of the present disclosure.

[0009] Figure 1 is a flowchart of an exemplary method for binocular disparity correction.

[0010] Figure 2 is a block diagram of an exemplary system for binocular disparity detection and correction.

[0011] Figure 3 is an illustration of an exemplary device having a binocular display and a disparity sensor.

[0012] Figures 4A to 4C is an illustration of an exemplary test pattern.

[0013] Figure 5 is an illustration of a device having a rolling display and a disparity camera.

[0014] Figure 6 is an illustration of multiple rows of a content frame.

[0015] Figure 7 is a synchronization diagram between a display and a sensor.

[0016] Figure 8 is an illustration of an exemplary closed-loop system for binocular disparity detection and correction.

[0017] Figure 9 is an illustration of an exemplary augmented reality glasses that can be used in conjunction with embodiments of the present disclosure.

[0018] Figure 10 is an illustration of an exemplary virtual reality headset that can be used in conjunction with embodiments of the present disclosure.

[0019] Figure 11 FIG. is a diagram of an exemplary system incorporating an eye tracking subsystem capable of tracking a user's monocular or binocular vision.

[0020] Figure 12 is Figure 11 a more detailed diagram of various aspects of the eye tracking subsystem shown in.

[0021] Throughout the drawings, the same reference numerals and descriptions indicate similar but not necessarily identical elements. Although the various exemplary embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. However, the various exemplary embodiments described herein are not intended to be limited to the particular forms disclosed. Rather, this disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims. DETAILED DESCRIPTION

[0022] Artificial reality devices typically employ a wearable form factor with near-eye optics to place a display near a user's eyes. For example, a user may wear glasses or other head-mounted devices that place a display, such as a waveguide display, near the user's eyes. The display may be positioned in a desired orientation with respect to the user's eyes to accurately display content to the user. However, due to various factors such as the user's head size, the user's body movement, tipping, tilting, etc., the display may deviate from the desired orientation. Such deviation may result in distortion, color change, lighting change, and / or other visual defects seen by the user.

[0023] Some artificial reality devices may provide correction for the foregoing visual defects. For example, a camera (e.g., a parallax camera) may capture the displayed content to identify visual defects and modify the content to account for the visual defects. However, when using a parallax camera, it may be challenging to identify visual defects and further determine how to modify the content to account for the visual defects.

[0024] The present disclosure generally relates to binocular disparity detection and correction. As will be explained in more detail below, embodiments of the present disclosure may acquire a first test pattern on a first display and a second test pattern on a second display. The systems and methods described herein may adjust the output to the first display and / or the second display by determining the deviation between the acquired first test pattern and the acquired second test pattern and the corresponding original first test pattern and original second test pattern to correct visual defects such as binocular disparity. Thus, embodiments of the present disclosure advantageously provide efficient and accurate binocular disparity correction. The systems and methods provided herein advantageously improve the functionality of a computer by reducing the arithmetic and computational resources required to detect and correct binocular disparity and other related visual defects. Additionally, the systems and methods provided herein advantageously improve the technical field of near-eye optics and waveguide displays.

[0025] In accordance with the general principles described herein, multiple features from any of the embodiments described herein may be used in combination with each other. These and other embodiments, features, and advantages will be more fully understood when reading the following detailed description in conjunction with the accompanying drawings and the claims.

[0026] A detailed description of binocular disparity detection and correction will be provided below with reference to Figures 1 to 12 A detailed description of example methods for binocular disparity detection and correction will be provided in conjunction with Figure 1 、 Figures 4A to 4C and Figure 8 A detailed description of example systems for binocular disparity detection and correction will be provided in conjunction with Figure 2 、 Figure 3 and Figure 8 A detailed description of example systems for binocular disparity detection and correction will be provided in conjunction with Figures 5 to 7 A detailed description of frame synchronization as used in binocular disparity detection / correction will be provided. Additionally, a detailed description of example devices will be provided in conjunction with Figures 9 to 12

[0027] Figure 1 is a flowchart of an exemplary computer-implemented method 100 for binocular disparity detection and correction. Figure 1 The steps shown in Figure 2 、 Figure 3 、 Figure 5 and / or Figure 8 shown in the system can be performed by any suitable computer-executable code and / or computing system (including Figure 1 shown in the system). In one example, each of the steps shown in

[0028] As Figure 1As shown, at step 102, one or more of the systems described herein may display a first test pattern on a first display and a second test pattern on a second display. For example, Figure 2 the pattern module 208 in

[0029] In some embodiments, the term "test pattern" may refer to an image or frame of a display that may include one or more visual features that can be relatively accurately identified (e.g., detected when viewing the test pattern and the position is determinable), and the position of the one or more visual features (e.g., relative to a reference coordinate system of the image) may be known. Examples of test patterns include, but are not limited to, dot patterns (e.g., having various dots at multiple different positions), line patterns, specific shapes located at specific positions, etc.

[0030] The various systems described herein may perform step 102. Figure 2 is a block diagram of an example system 200 for binocular disparity detection and correction. As shown in the figure, the example system 200 may include one or more modules 202 for performing one or more tasks. As will be explained in more detail herein, the module 202 may include a synchronization module 204, a detection module 206, a pattern module 208, and an adjustment module 210. Although Figure 2 one or more of these modules 202 in

[0031] In certain embodiments, Figure 2 one or more of these modules 202 in Figure 5 may represent one or more software applications or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks. For example, as will be described in more detail below, one or more of these modules 202 may represent the following modules: these modules are stored on one or more computing devices and are configured to run on the one or more computing devices, such as the Figure 2 devices shown in

[0032] As in Figure 2As shown, example system 200 may also include one or more storage devices, such as memory 240. Memory 240 generally represents any type or form of volatile or non-volatile storage device or storage medium capable of storing data and / or computer-readable instructions. In one example, memory 240 may store, load, and / or maintain one or more of these modules 202. Examples of memory 240 include, but are not limited to, random access memory (RAM), read only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), optical disk drive, cache memory, variations or combinations of one or more of the above, and / or any other suitable storage memory.

[0033] As Figure 2 shown, example system 200 may also include one or more physical processors, such as physical processor 230. Physical processor 230 generally represents any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, physical processor 230 may access and / or modify one or more of these modules 202 stored in memory 240. Additionally or alternatively, physical processor 230 may execute one or more of these modules 202 to facilitate a mapping system. Examples of physical processor 230 include, but are not limited to, microprocessors, microcontrollers, central processing unit (CPU), field-programmable gate array (FPGA) implementing a soft-core processor, application-specific integrated circuit (ASIC), portions of one or more of the above, variations or combinations of one or more of the above, and / or any other suitable physical processor.

[0034] As Figure 2As shown, the example system 200 may also include one or more additional elements 220, such as a display 222, a camera 224, a test pattern 252, a captured test pattern 254, and an adjusted frame 256. The test pattern 252, the captured test pattern 254, and / or the adjusted frame 256 may be stored on a local storage device (e.g., the memory 240), or may be accessed remotely. The display 222 may represent any display device, including one or more display devices such as a liquid crystal on silicon (LCOS) display and / or a micro-LED-based display, as will be further described below. The camera 224 may represent any optical sensor, such as a rolling shutter camera and / or a global shutter camera, and may also include additional components as will be further described below. The test pattern 252 may correspond to one or more visual patterns (e.g., a separate visual pattern for each display), which may be generated using specific visual features for detecting binocular disparity and / or other visual distortions, as will be further described below. The captured test pattern 254 may represent the image data of the test pattern 252 captured by the camera 224 as displayed on the display 222, as will be further described below. The adjusted frame 256 may represent one or more frames rendered for display on the display 222, which may incorporate binocular disparity correction and / or other visual defect correction before being displayed by the display 222, as will be further described below.

[0035] Figure 2 The example system 200 in can be implemented in a variety of different ways. For example, all or a portion of the example system 200 may represent one or more communicatively coupled computing devices. Figure 3 An example device 300 corresponding to the system 200 is shown.

[0036] Figure 3 An example device 300 is shown that may correspond to a binocular display device, such as a head-mounted display (HMD) or a near-eye display having one display for each eye. The device 300 includes a display 322A (e.g., a first display or left display for the left eye) and a display 322B (e.g., a second display or right display for the right eye). In some embodiments, the display 322A and / or the display 322B may correspond to a waveguide (e.g., a structure that guides light frequencies by total internal reflection at an interface between media having different refractive indices). For example, Figure 3 Generally shows how light waves can be reflected.

[0037] As Figure 3 further shown in Figure 3 , projector 323A (e.g., the first projector or the left projector) can project light waves onto display 322A, and projector 323B (e.g., the second projector or the right projector) can project light waves onto display 322B. Projectors 323A and 323B can project different frames based on binocular vision (e.g., for each eye). Thus, displays 322A, projector 323A, display 322B, and projector 323B can together correspond to display 222. In Figure 3 Figure 3 , display 322A and projector 323A can be integrated into the left side of the frame (see, e.g., Figure 9 left display 915(A) in Figure 9 ), and display 322B and projector 323B can be integrated into the right side of the frame (see, e.g., Figure 9 right display 915(B) in Figure 9 ). In other embodiments, display 322A and / or display 322B can correspond to other types of displays, which may also not use the corresponding projectors (e.g., projector 323A and / or projector 323B).

[0038] Device 300 further includes a parallax camera 324 and a parallax sensor waveguide 325. Parallax camera 324 can correspond to an optical sensor for collecting image data from parallax sensor waveguide 325. Parallax sensor waveguide 325 can correspond to a waveguide configured to propagate light waves from displays 322A and 322B. As Figure 3 shown, the left-eye image displayed by display 322A can propagate from the left side of parallax sensor waveguide 325 to the center of parallax sensor waveguide 325, and then the left-eye image can be focused by a lens or other optical element and collected by parallax camera 324. Similarly, the right-eye image displayed by display 322B can propagate from the right side of parallax sensor waveguide 325 to the center of parallax sensor waveguide 325, and then the right-eye image can be focused by a lens / optical element and collected by parallax camera 324. Thus, parallax camera 324 and parallax sensor waveguide 325 can together correspond to camera 224.

[0039] In Figure 3 Figure 3 , parallax camera 324 and parallax sensor waveguide 325 can be integrated into the nose bridge of the frame to be centered between displays 322A and 322B and collect both displays. Although having a parallax camera 324 that can collect both displays can reduce the components required in the frame and reduce design complexity, other configurations can be used in other embodiments, such as multiple cameras, cameras located at different positions along the frame, etc.

[0040] Returning toFigure 1 For step 102, display 322A may display a first test pattern, and display 322B may display a second test pattern. Figure 4A and Figure 4B illustrates example test patterns. Figure 4A illustrates test pattern 452A (e.g., a left test pattern that may be displayed on a left display (e.g., display 322A)), and Figure 4B illustrates test pattern 452B (e.g., a right test pattern that may be displayed on a right display (e.g., display 322B)). As Figure 4A and Figure 4B shown, test pattern 452A and test pattern 452B may each correspond to a different repetition of the dot pattern. Although Figure 4A and Figure 4B illustrates a simplified example of a test pattern, in other examples, other test patterns may be used, such as those having more or fewer dots, different shapes and / or colors, gradients, and / or other distinguishable visible features.

[0041] Although test pattern 452A and test pattern 452B may correspond to predefined test patterns, in some embodiments, test pattern 452A and / or test pattern 452B may be dynamically generated as needed, e.g., by pattern module 208. Test pattern 452A may be different from test pattern 452B, and more specifically may be high-probabilistically different. For example, pattern module 208 may randomly or pseudo-randomly determine the positions of the dots in test pattern 452A and / or test pattern 452B, but may select positions that have a high-probability difference between the two test patterns to reduce the likelihood that a dot in test pattern 452A is near (e.g., within a statistical distance threshold) a dot in test pattern 452B in the same coordinate system (e.g., frame), as will be further explained below.

[0042] Returning to Figure 1 , the systems described herein may perform step 102 in a variety of different ways. In some examples, the test patterns may be displayed in a manner that reduces a user's notice of the test patterns. In some examples, eye tracking (as will be described below with reference to Figure 11 and Figure 12Further explanations and / or other user detections can be used to display the test pattern in an unobtrusive manner. For example, the first test pattern and / or the second test pattern can be displayed when a user blink is detected, and / or when it is detected that the user's gaze is away from at least one of the displays in the display. In other examples, the first test pattern and / or the second test pattern can be hidden in the output frame so that the user can normally view the content, and the test pattern features can be embedded in the output frame (e.g., in a specific color and / or color channel) so as not to detract from the content but remain recognizable. In still other examples, the first test pattern can be displayed asynchronously with the second test pattern.

[0043] Continue Figure 1 At step 104, one or more of the systems described herein can collect the first test pattern displayed and the second test pattern displayed. For example, the camera 224 can collect the test pattern 252 displayed on the display 222 as the collected test pattern 254.

[0044] To collect the displayed frame and correctly associate the collected frame with the rendered frame, the synchronization module 204 can implement a timing scheme. In one example, the timing scheme can correspond to a time synchronization protocol such as the precision time protocol (PTP). The timing scheme can be implemented using a time synchronizer (TS) module (e.g., Figure 5 the TS 504 in

[0045] Figure 5System 500 is shown, which may include a host 570, a parallax camera 524 (which may correspond to camera 224), and a display 522 (which may correspond to display 222). The host 570 may be a computing device that interfaces with the parallax camera 524 via a camera controller 574 and with the display 522 via a display controller 572. The host 570 may include host software 576, which may correspond to artificial reality software that interfaces with the camera controller 574 and / or the display controller 572 via module 202. These modules 202 may include a camera ingest 514, a display driver 512, and a time synchronizer 504. The camera ingest 514 may correspond to software and / or hardware modules for receiving image data from the parallax camera 524 and otherwise interfacing with the parallax camera 524. The display driver 512 may correspond to software and / or hardware modules for configuring the display 522 and otherwise interfacing with the display 522. The time synchronizer 504 (which may correspond to synchronization module 204) may correspond to software and / or hardware modules for implementing a timing scheme for the parallax camera 524 and the display 522.

[0046] The parallax camera 524 may correspond to an image sensor that may be oriented to collect image data from the display 522, such as a rolling shutter camera, a global shutter camera, etc. The camera controller 574 may provide an interface for controlling the parallax camera 524 and implement TS 504. In some examples, the parallax camera 524 may be a rolling shutter camera that is capable of collecting an image (and / or a series of images / frames of video) by scanning across the image rather than scanning the entire image at once. In some examples, the parallax camera 524 may be a global shutter camera that is capable of collecting the entire image in one instance.

[0047] The display 522 may correspond to a rolling display, a waveguide, and / or other displays such as a micro-LED display. The display controller 572 may provide an interface for controlling the display 522 and implement TS504. In some examples, the parallax camera 524 may be a rolling display that is capable of displaying an image (and / or a series of images / frames of video) by displaying the image across rather than displaying the entire image at once.

[0048] In some examples, synchronizing the parallax camera 524 and the display 522 may further include: configuring the parallax camera 524 and / or the display 522 to match a timing window for displaying a frame (and / or a portion thereof) with a timing window for acquiring a frame (and / or a portion thereof). For example, the camera imager 514 and / or the camera controller 574 may set the frame rate (e.g., frames per second) of the parallax camera 524 to twice the display rate of the display 522 (which may be set by the display driver 512 and / or the display controller 572), such that the parallax camera 524 may acquire frames at twice the rate at which the display 522 displays each frame.

[0049] In some examples, the camera imager 514 and / or the camera controller 574 may set the exposure time of the parallax camera 524 to match the display duration of the display 522 (which may be set by the display driver 512 and / or the display controller 572). For example, the exposure time of the parallax camera 524 may be set to be less than or equal to the display duration (e.g., the amount of time for which the display 522 displays a frame or a portion thereof). In some examples, the exposure time may be set to be greater than or exceed the display duration. The line interval time for the display 522 (e.g., the amount of time between the display of one line of a frame and the display of the next line of the frame by the display 522) may match the line interval time of the parallax camera 524 (e.g., the amount of time between the acquisition of one line of a frame and the acquisition of the next line of the frame by the parallax camera 524). The line interval time may also correspond to the scroll start window (e.g., the amount of time for which the parallax camera 524 traverses each line). For example, the scroll start window may be equal to the line interval time multiplied by the number of lines.

[0050] Figure 6 A diagram of a frame 600 that may be displayed by a scrolling display is further shown. Instead of displaying a single complete frame at a time, a scrolling display may asynchronously display portions of a frame, such as traversing the portions (which may overlap) over time until the entire frame has been displayed. For example, the frame 600 may be divided into multiple lines (e.g., lines 1 to N).

[0051] In some embodiments, the term "row" may refer to a portion of a frame (e.g., an image that may be part of a sequence of images for a video). For example, if a frame is represented as a matrix of pixel (color) values, a row as used herein may refer to a row and / or column of the matrix. In some examples, a row may refer to a portion of a row and / or column of the matrix, multiple rows and / or columns, or other subsets of the matrix. In some examples, a row may refer to an entire frame (e.g., the entire matrix). In some examples, a row may refer to a portion of a frame (e.g., a rendered image) that is being displayed or will be displayed, and / or may directly refer to a corresponding display area of a display. In some examples, a row may refer to a portion of a frame (e.g., a frame captured by an image sensor) that will be captured, and / or may directly refer to a corresponding subset of sensors from an image sensor.

[0052] As Figure 6 shown, the display duration 680 may refer to the length of time that a given row is displayed by a scrolling display such as display 522. In Figure 6 this, although the display duration 680 may be uniform or otherwise consistent for all rows (rows 1 to N), in other examples, the display duration 680 may vary between rows. Additionally, Figure 6 shown is an inter-row time 682 corresponding to the time delay between displaying one row and displaying its adjacent row. Although Figure 6 shown is a uniform or otherwise consistent inter-row time 682 between all rows (rows 1 to N), in other examples, the inter-row time 682 may vary.

[0053] Accordingly, a scrolling display such as display 522 may display frame 600 as a series of rows 1 to N that are out of sync. Thus, a camera such as parallax camera 524 may need to be synchronized with display 522 to appropriately capture frame 600. For example, a global shutter camera may need to capture frame 600 at the moment when all rows (rows 1 to N) are being displayed simultaneously. A rolling shutter camera may capture rows 1 to N while rows 1 to N are being displayed.

[0054] Triggers can be used to control the timing of frame display and acquisition. In one example, the display controller 572 can trigger the display 522 to display the content frame 552 via a display trigger 562. The display 522 can accordingly display the content frame 552, and in some examples, the content frame 552 can correspond to the test pattern 252. In some examples, the display 522 can display the content frame 552 row by row. The TS 504 can send the display trigger 562 to the display 522 based on a timing scheme (via the display controller 572). For example, the display trigger 562 can correspond to the modulo value of the timestamp value of the timing scheme. In some examples, a modulo value can be specified for the display trigger 562. Additionally, in some examples, the display controller 572 can save the display timestamp corresponding to triggering the display 522 to display the content frame 552. In some examples, the display timestamp can correspond to the display trigger 562 (e.g., the timestamp value associated with the display trigger 562).

[0055] Figure 7 A timing diagram 700 showing the display and acquisition of each row (e.g., rows 1 to N as in Figure 6 ) is shown. The x-axis can correspond to time and the y-axis can correspond to rows. A display such as the display 522 can display row 1 in response to a display trigger 762 which can correspond to the display trigger 562. As shown, the display can continue to display rows 2 to N over time. The roll start window 784 can correspond to the time elapsed between the start of displaying row 1 and the start of displaying row N. The duration 780 can correspond to the length of time a frame or row is displayed. The period 782 can correspond to the frequency at which a frame is displayed (e.g., a period of 11 ms before displaying the next frame can correspond to 90 frames per second (fps)). The display roll-out 792 ( Figure 7 shown as the non-contiguous shaded portion in ) can correspond to the display time of a frame (more specifically, each row from row 1 to row N). The display roll-out 792 shows that each row from row 1 to row N is displayed for an amount of time corresponding to the duration 780, and after starting to display row 1, row N can stop being displayed at a time approximately equal to the sum of the duration 780 and the roll start window 784. Additionally, although not shown in Figure 7 , in some examples, each row from row 1 to row N can be triggered by a corresponding display trigger 762, and a corresponding display timestamp can be saved for each row.

[0056] For a capture frame, the camera controller 574 can trigger the disparity camera 524 via the capture trigger 564 to capture a captured content frame 554 (which can correspond to a captured test pattern 254 in one example) from the display 522. In some examples, the disparity camera 524 can capture the captured content frame 554 row by row. The TS 504 can send the capture trigger 564 to the disparity camera 524 based on a timing scheme (via the camera controller 574). For example, the capture trigger 564 can correspond to the modulo value of the timestamp value of the timing scheme. In some examples, a modulo value can be specified for the capture trigger 564, and the modulo value can be selected to have an offset from the modulo value of the display trigger 562. Additionally, in some examples, the camera controller 574 can save a capture timestamp that corresponds to triggering the disparity camera 524 to capture the captured content frame 554. In some examples, the capture timestamp can correspond to the capture trigger 564 (e.g., the timestamp value associated with the capture trigger 564).

[0057] Figure 7 A capture trigger 764 that can correspond to the capture trigger 564 is shown. As Figure 7 shown, the capture trigger 764 can be offset from the display trigger 762. The sensor exposure 794 shows how a rolling shutter camera (e.g., the disparity camera 524) can capture frames row by row (e.g., rows 1 to N). The exposure time 786 can correspond to the duration for which a frame or a row is captured by the corresponding sensor. Thus, the sensor exposure 794 shows how the rows of a frame can be captured. Additionally, as Figure 7 shown, the sensor exposure 794 can not exceed the display advancement 792 (e.g., not extend beyond the boundaries of the display advancement), such that the camera is capturing the displayed frame without capturing irrelevant image data. The offset of the capture trigger 764 and / or the exposure time 786 can be selected accordingly to ensure that the sensor exposure 794 does not exceed the display advancement 792. For example, the exposure time 786 can be less than the duration 780.

[0058] In other examples, based on the type of camera, the sensor exposure 794 can have a different shape, such as a rectangle corresponding to a global shutter camera (e.g., by capturing all rows (rows 1 to N) in the same time period). In such examples, the offset of the capture trigger 764 and / or the exposure time 786 can be selected accordingly to ensure that the sensor exposure 794 can not exceed the display advancement 792. Additionally, although not shown in Figure 7 it, in some examples, each of rows 1 to N can be triggered by a corresponding capture trigger 764, and a corresponding capture timestamp can be saved for each row.

[0059] In some examples, unwanted visual effects in the captured content frames (captured test pattern 254) may be attributed to noise such as ambient light and / or other light sources. To detect the background light and distinguish it from the light from the display 222, the camera 224 may capture a background light frame 256 corresponding to the background light when the display is not actively displaying content. In some examples, the ambient light may be passively filtered, such as a coating on the lens of the camera 224 and / or other filters in front of the lens.

[0060] For example, the camera controller 574 may trigger the parallax camera 524 to capture the captured background light frame via a background light trigger 566. In some examples, the parallax camera 524 may capture the captured background light frame row by row. The TS 504 may send the background light trigger 566 to the parallax camera 524 based on a timing scheme (via the camera controller 574). For example, the background light trigger 566 may correspond to the modulo value of the timestamp value of the timing scheme. In some examples, a modulo value may be specified for the background light trigger 566, which may be selected to have an offset from the modulo values of the capture trigger 564 and / or the display trigger 562. Additionally, in some examples, the camera controller 574 may save a background light timestamp corresponding to triggering the parallax camera 524 to capture the captured background light frame. In some examples, the background light timestamp may correspond to the background light trigger 566 (e.g., the timestamp value associated with the background light trigger 566).

[0061] Figure 7 A background light trigger 766 that may correspond to the background light trigger 566 is shown. As Figure 7 shown, the background light trigger 766 may be offset from the capture trigger 764. The sensor exposure 796 shows how a rolling shutter camera (e.g., the parallax camera 524) may capture frames row by row (e.g., rows 1 to N). The settings of the sensor exposure 796 may be similar to the settings of the sensor exposure 794, as reflected by their similar shapes. Thus, the sensor exposure 796 shows how the rows of the background light frame may be captured (e.g., outside of the display advancement 792). The offset of the background light trigger 766 may be selected accordingly to ensure that the sensor exposure 796 does not overlap with the display advancement 792. The offset may be based on, for example, half of the period 782 or other appropriate values corresponding to when the display is not actively displaying content frames. Additionally, although Figure 7 shown the background light trigger 766 is performed after the display has finished displaying the corresponding content frame, in some examples, the background light trigger 766 may be performed before the display begins to display the corresponding content frame.

[0062] In addition, although not shown in Figure 7 in some examples, the background light for each of the acquisition rows 1 to N can be triggered by a corresponding background light trigger 766, and a corresponding background light timestamp can be saved for each row.

[0063] To organize the frames, the synchronization module 204 can construct a timeline of frame events for the displayed content frames, the acquired content frames, and in some examples the acquired background light frames. In one example, the synchronization module 208 can receive the displayed content frame (e.g., test pattern 252), the acquired content frame (e.g., acquired test pattern 254), and / or the acquired background light frame and their respective timestamps (e.g., the display timestamp, the acquisition timestamp, and the background light timestamp as described above). For example, the camera controller 574 can send the acquired frame 578 (which can include the acquired content frame 554 and the acquired background light frame) along with the acquisition timestamp and the background light timestamp to the camera imager 514. The display controller 572 can send the display timestamp to the host 570 to allow the host 570 to construct a timeline of frame events.

[0064] In some examples, the synchronization module 208 can include a state machine such that constructing the timeline of frames 250 can include inputting the display timestamp, the acquisition timestamp, and the background timestamp into the state machine.

[0065] In some examples, the timeline can be more fine-grained than the frames and can alternatively or additionally include a timeline of row events. For example, each row can have a display timestamp, an acquisition timestamp, and a background light timestamp such that the timeline can organize the timestamps based on rows.

[0066] Using the timeline, the synchronization module 204 can match the displayed content frame (e.g., test pattern 252) with the corresponding acquired content frame (e.g., acquired test pattern 254) and the acquired background light frame (if available). For example, the synchronization module 204 can match the test pattern 252 with the acquired test pattern 254, where the offset between the acquisition timestamp of the acquired test pattern and the display timestamp is up to the expected offset between the display timestamp and the acquisition timestamp (the display timestamp and the acquisition timestamp can correspond to respective modulo values). Similarly, if appropriate, the synchronization module 204 can offset the acquired background light frame based on the background light timestamp of the acquired background light frame by the expected offset between the background light timestamp and the acquisition timestamp (and / or the display timestamp) and match the acquired background light frame with the acquired test pattern 254 (and / or the test pattern 252). In some examples, the background light timestamp and the acquisition timestamp (and / or the display timestamp) can be based on respective modulo values.

[0067] In some examples, the synchronization module 204 may construct a timeline by matching corresponding displayed content frames, captured content frames, and / or captured background light frames to match the frames using the organization and / or sequence of frame events. Additionally, in some examples, the synchronization module 204 may match each row (e.g., corresponding rows from test pattern 252, captured test pattern 254, and / or captured background light frames). Thus, the synchronization module 204 may allow captured frames to be matched to corresponding displayed / rendered frames.

[0068] Returning to Figure 1 , the systems described herein may perform step 104 in a variety of ways. In one example, the first test pattern displayed and the second test pattern displayed may be captured simultaneously in a combined image. For example, the disparity camera 324 may capture test patterns from displays 322A and 322B as a fused dot pattern of two different dot patterns, an example of which is shown in Figure 4C FIG.

[0069] Figure 4C FIG. shows a fused test pattern 454 (which may correspond to the captured test pattern 254). As shown in Figure 4C FIG., the fused test pattern 454 may correspond to a superimposed combination of test pattern 452A and test pattern 452B, further corresponding to how the disparity camera 324 may capture test pattern 452A and test pattern 452B. The pattern module 208 may generate different test patterns for each display (e.g., test pattern 452A and test pattern 452B) such that the different test patterns (e.g., in the fused test pattern 454) can be identified even when overlapping. Additionally, although Figures 4A to 4C FIG. shows visually distinct shapes for each test pattern (e.g., dots with different shadings for each of test pattern 452A and test pattern 452B) to show the fused test pattern (e.g., the fused test pattern 454 of dots with two shading patterns), in other examples, the visual features may appear similar and be in different positions.

[0070] In addition, in some embodiments, the acquired test pattern 254 can be processed to reduce noise or otherwise improve visual quality to facilitate visual feature detection. For example, background light and other noise related to ambient light can be removed (e.g., by using the background light frames described herein) from the acquired test pattern 254. In some examples, if the test pattern 252 is embedded or hidden in a content frame before being displayed on the display 222, for example, the image data corresponding to the content frame can be removed from the acquired test pattern 254 such that the visual features of the test pattern 252 are retained in the acquired test pattern 254 when acquired.

[0071] Continuing Figure 1 , at step 106, one or more of the systems described herein can determine a deviation between at least one of the following: the acquired first test pattern and the first test pattern, or the acquired second test pattern and the second test pattern. For example, the detection module 206 can determine one or more visual feature deviations between the acquired test pattern 254 and the test pattern 252.

[0072] In some embodiments, the term "deviation" can refer to a mathematically and / or statistically significant difference between corresponding values, such as differences in position / orientation as will be further explained below.

[0073] The systems described herein can perform step 106 in a variety of ways. In one example, determining the deviation further includes distinguishing the acquired first test pattern from the acquired second test pattern in the combined image. The detection module 206 can identify a first set of expected features from the first test pattern and a second set of expected features from the second test pattern from the test pattern 252. For example, the detection module 206 can identify a first set of expected features from the test pattern 452A and a second set of expected features from the test pattern 452B. In some examples, the detection module 206 can identify the first set of expected features by constructing a point model based on the feature positions from the first test pattern (e.g., a model of the dot positions of the dots in the test pattern 452A), and similarly identify the second set of expected features by constructing a second point model based on the feature positions from the second test pattern (e.g., a model of the dot positions of the dots in the test pattern 452B). The position values can be based on a coordinate system that can correspond to pixel positions or other suitable reference coordinate systems. In addition, in some embodiments, the detection module 206 and / or the pattern module 208 can construct the point models when the pattern module 208 generates the corresponding test patterns. In other embodiments, the detection module 206 can identify the expected features from the rendered frames to be displayed on the display 222.

[0074] The detection module 206 can also identify a plurality of observed features from the combined image (e.g., the acquired test pattern 254). The detection module 206 can identify the plurality of observed features by determining the positions of the identified features from the combined image. More specifically, the detection module 206 can detect visual features based on feature detection and / or other computer vision, and / or using blink detection or a similar eye tracking process, which can include identifying visually distinguishable pixels (which can differ from adjacent pixels by at least a threshold in color and / or other values) from the image data of the acquired test pattern 254 and determining the corresponding positions (e.g., by calculating appropriate center points of the visually distinguishable pixels). For example, the detection module 206 can detect each dot in the fused test pattern 454. Although in some embodiments, the detection module 206 can further distinguish these different types of dots when, for example, different types of dots differ in style / color, in other embodiments, the dots can be similar in style such that the detection module 206 can identify each dot without distinguishing the types of dots. Additionally, in some embodiments, each display can be acquired separately (e.g., via a separate camera for each display or a camera that acquires one display at a time), such that the detection module 206 does not actively identify between a first set of observed features and a second set of observed features.

[0075] Accordingly, the detection module 206 can determine that a first set of observed features from the plurality of observed features corresponds to a first test pattern based on similarity to a first set of expected features, and can further determine that a second set of observed features from the plurality of observed features corresponds to a second test pattern based on similarity to a second set of expected features. For example, the detection module 206 can use a first dot model corresponding to the test pattern 452A and match the nearest dot detected in the fused test pattern 454 to determine the first set of observed features. Similarly, the detection module 206 can use a second dot model corresponding to the test pattern 452B and match the nearest dot detected in the fused test pattern 454 to determine the second set of observed features. In some examples, making the positions of the dots in the test pattern 452A likely to be different from the positions of the dots in the test pattern 452B can reduce the likelihood of a dot being associated with the wrong test pattern.

[0076] In some examples, the detection module 206 may select a first set of observed features from multiple observed features based on a relationship matrix of points between a point model and the first set of observed features, and further distinguish the first acquired test pattern and the second acquired test pattern based on a relationship matrix between a second point model and a second set of observed features. For example, the detection module 206 may generate a relationship matrix that may indicate the relationship (e.g., distance) between the dot points observed in the fused test pattern 454 and the dot points in the test pattern 452A and the test pattern 452B, such that the detection module 206 may use the relationship matrix to distinguish the dot points corresponding to the test pattern 452A or the test pattern 452B.

[0077] The detection module 206 may determine a first deviation for the first display based on comparing the first set of observed features with a first set of expected features. For example, the detection module 206 may use the relationship matrix to determine the deviation as a first transformation matrix (e.g., corresponding to rotation, scaling, shearing, reflection, orthogonal projection, etc.) between the position of the observed features and the expected features. The detection module 206 may similarly determine a second deviation (e.g., a second transformation matrix) for the second display based on comparing the second set of observed features with a second set of expected features.

[0078] At step 108, one or more of the systems described herein may adjust the output to at least one of the first display or the second display based on the determined deviation. For example, the adjustment module 210 may adjust the rendered frame to an adjusted frame 256 to be output to the display 222.

[0079] The systems described herein may perform step 108 in various ways. In one example, the adjustment module 210 may adjust the rendered frame provided by the host software 576. In some examples, the adjustment module 210 may adjust the output by applying a first correction to the first output of the first display based on the first deviation, and similarly applying a second correction to the second output of the second display. These corrections may be based on the point relationship matrix. For example, the correction may correspond to the deviation and may also correspond to the transformation matrix as described above. The first correction may correspond to the first transformation matrix, which when applied to the fused test pattern 454 will cause the dot points corresponding to the test pattern 452A to be positioned in the test pattern 452A. The second correction may correspond to the second transformation matrix, which when applied to the fused test pattern 454 will cause the dot points corresponding to the test pattern 452B to be positioned in the test pattern 452B.

[0080] In other words, the adjustment module 210 can apply a first transformation matrix to the frames of the first display (e.g., display 322A) to reverse the calculated first deviation between the acquired first test pattern and the original first test pattern, and can apply a second transformation matrix to the frames of the second display (e.g., display 322B) to reverse the calculated second deviation between the acquired second test pattern and the original second test pattern. Additionally, in some embodiments, the adjustment module 210 can selectively apply corrections to the displays and / or frames. For example, the adjustment module 210 can determine that there is no significant deviation (e.g., the corresponding transformation matrix may not meet the transformation threshold). Additionally, one or more of steps 102, 104, 106, and 108 can be repeated as needed so that the test patterns and / or corrections can be updated dynamically, and the adjustment module 210 can apply the corrections accordingly as needed.

[0081] In some embodiments, the binocular disparity detection / estimation and correction system described herein can correspond to a closed-loop system. Figure 8 A data flow diagram of system 800 is shown. System 800 can correspond to system 200, device 300, system 500, and / or any other system or device described herein. System 800 can conceptually be divided into a user application 876 (e.g., software or other user application that can generate content frames, such as host software 576) and a closed-loop sensing system 801. The user application 876 can include a display image 877 (e.g., a rendered content frame from the user application 876).

[0082] As Figure 8 shown, the closed-loop sensing system 801 can include disparity sensing 806 (corresponding to detection module 206), display correction 810 (corresponding to adjustment module 210), a data synchronization system 804 (corresponding to synchronization module 204), a camera frame 854 (in some instances, corresponding to the acquired test pattern 254), a display frame 852 (in some instances, corresponding to test pattern 252), a display rendering pipeline 811 (partially corresponding to display controller 572), and a corrected frame 856 (corresponding to the adjusted frame 256).

[0083] The camera can capture camera frames 854 with timestamps (as described herein), e.g., as in step 104, for matching with corresponding display frames 852 with timestamps (as described herein), e.g., as in step 102. In some examples, the display frame 852 can correspond to a display image 877, and more specifically, to a previous iteration or instance of the display image 877. The data synchronization system 804 can build a timeline (as described herein) to match the camera frames 854 with the display frames 852 for analysis by the disparity sensing 806 (e.g., as in step 106).

[0084] The camera frames 854 and the display frames 852 can include test patterns (e.g., see Figures 4A to 4C ), such that the disparity sensing 806 can estimate binocular disparity and send display correction information to the display correction 810. The display correction 810 can apply the display correction to the display image 877 based on the received display correction information (e.g., as in step 106). In some examples, the display image 877 can correspond to data for rendering a frame, such that applying the display correction can change the data. Then, the display rendering pipeline 811 can use the changed data to render the frame as a corrected frame 856 for display on a display (e.g., display 222). Additionally, Figure 8 the processes described in

[0085] can be applied to multiple displays, which can be applied to multiple displays by processing frames in parallel (e.g., having parallel closed-loop sensing systems 801 and / or portions thereof for each display) and / or interleaved (e.g., the closed-loop sensing system 801 provides separate corrections for frames intended for different displays). Figure 4A and Figure 4B ) can be periodically displayed on the projectors. Then, the fused dot pattern (see Figure 4C ) is captured by the disparity sensor. Given the fused dot pattern image captured by the disparity sensor, the systems and methods described herein estimate the amount of binocular disparity introduced by frame distortion.

[0086] For example, as described herein, a fused dot pattern image (or disparity signal) collected by a disparity sensor can be provided to a detection module (e.g., detection module 206) to locate the dots. In some examples, a flicker detection algorithm similar to that used in eye tracking applications can be used to locate flicker in the human eye. Once the dot locations are determined, the groups of dots from the left and right displays are separated.

[0087] In some examples, the detection module can use a random sample consensus algorithm (RANSAC), which can be used to separate the left disparity signal from the right disparity signal. Given the expected positions of all dots from the left display, the detection module can use RANSAC to identify the spatially perturbed versions of these dots as inliners and the dots from the right display as outliers. Therefore, the detection module using RANSAC can effectively separate the dots from the two displays. During this separation process, the detection module can also use RANSAC to generate a misalignment between the separated disparity signals and their expected positions (assuming there is no binocular parallax relative to the rendered frame). This misalignment corresponds to the movement between the projector and the main waveguide described herein. Once the misalignment is known, the graphics pipeline can effectively correct future images to compensate for the misalignment.

[0088] Example Embodiments

[0089] Example 1: A computer-implemented method comprising: displaying a first test pattern on a first display and displaying a second test pattern on a second display; acquiring the displayed first test pattern and the displayed second test pattern; determining a deviation between at least one of: the acquired first test pattern and the first test pattern, or the acquired second test pattern and the second test pattern; and adjusting an output to at least one of the first display or the second display based on the determined deviation.

[0090] Example 2: The method according to Example 1 further includes: simultaneously capturing a displayed first test pattern and a displayed second test pattern in a combined image, wherein the first test pattern is likely to be different from the second test pattern.

[0091] Example 3: The method of Example 2, wherein determining the deviation further comprises: distinguishing between the acquired first test pattern and the acquired second test pattern in the combined image.

[0092] Example 4: The method according to Example 3, wherein differentiating the first test pattern collected from the second test pattern collected further comprises: identifying a first set of expected features from the first test pattern and a second set of expected features from the second test pattern; identifying a plurality of observed features from the combined image; and determining that a first set of observed features from the plurality of observed features corresponds to the first test pattern based on similarity to the first set of expected features, and determining that a second set of observed features from the plurality of observed features corresponds to the second test pattern based on similarity to the second set of expected features.

[0093] Example 5: The method according to Example 4, wherein: determining the deviation further comprises: determining a first deviation for the first display based on comparing the first set of observed features with the first set of expected features; and adjusting the output further comprises: applying a first correction to the first output of the first display based on the first deviation.

[0094] Example 6: The method according to Example 5, wherein: identifying the first set of expected features comprises constructing a point model based on the feature positions from the first test pattern; identifying a plurality of observed features comprises determining the positions of the identified features from the combined image; differentiating the first test pattern collected from the second test pattern collected further comprises: selecting a first set of observed features from the plurality of observed features based on a point relationship matrix between the point model and the first set of observed features; and determining the first deviation is based on the point relationship matrix.

[0095] Example 7: The method according to Example 6, wherein the first correction is based on the point relationship matrix.

[0096] Example 8: The method according to any one of Examples 1 to 7, further comprising: displaying at least one of the first test pattern or the second test pattern when it is detected that the user blinks.

[0097] Example 9: The method according to any one of Examples 1 to 8, further comprising: displaying at least one of the first test pattern or the second test pattern when it is detected that the user's gaze is away from at least one of the first display or the second display.

[0098] Example 10: The method according to any one of Examples 1 to 9, further comprising: displaying at least one of the first test pattern or the second test pattern hidden in the output frame.

[0099] Example 11: The method according to any one of Examples 1 to 10, further comprising: displaying the first test pattern and the second test pattern asynchronously.

[0100] Example 12: A system comprising: at least one physical processor; a first display; a second display; an image sensor device; and a physical memory, the physical memory comprising computer executable instructions that, when executed by the physical processor, cause the physical processor to: display a first test pattern on the first display and display a second test pattern on the second display; capture the displayed first test pattern and the displayed second test pattern using the image sensor device; determine a deviation between at least one of: the captured first test pattern and the first test pattern, or the captured second test pattern and the second test pattern; and adjust the output to at least one of the first display or the second display based on the determined deviation.

[0101] Example 13: The system of Example 12, wherein the image sensor device is configured to simultaneously capture a displayed first test pattern and a displayed second test pattern in a combined image, wherein the first test pattern is likely to be different from the second test pattern.

[0102] Example 14: A system according to Example 13, wherein the instructions for determining the deviation further include: instructions for distinguishing a captured first test pattern and a captured second test pattern in a combined image by the following steps: identifying a first set of expected features from the first test pattern and a second set of expected features from the second test pattern; identifying multiple observation features from the combined image; and determining that a first set of observation features from the multiple observation features corresponds to the first test pattern based on similarity to the first set of expected features, and determining that a second set of observation features from the multiple observation features corresponds to the second test pattern based on similarity to the second set of expected features.

[0103] Example 15: A system according to Example 14, wherein: the instructions for determining a deviation further include: instructions for determining a first deviation for a first display based on comparing a first set of observed features and a first set of expected features; and the instructions for adjusting the output further include: instructions for applying a first correction to a first output of the first display based on the first deviation.

[0104] Example 16: A system according to Example 15, wherein: instructions for identifying a first set of expected features include instructions for constructing a point model based on feature locations from a first test pattern; instructions for identifying multiple observed features include instructions for determining locations of the identified features from a combined image; instructions for distinguishing between a captured first test pattern and a captured second test pattern further include: instructions for selecting a first set of observed features from multiple observed features based on a point relationship matrix between the point model and the first set of observed features; determining the first deviation is based on the point relationship matrix; and the first correction is based on the point relationship matrix.

[0105] Example 17: A system according to any one of Examples 12 to 16, wherein the instructions for further displaying the first test pattern and the second test pattern include at least one of the following: instructions for displaying at least one of the first test pattern or the second test pattern when a blink of the user is detected; instructions for displaying at least one of the first test pattern or the second test pattern when a user is detected looking away from at least one of the first display or the second display; instructions for displaying at least one of the first test pattern or the second test pattern hidden in an output frame; and instructions for displaying the first test pattern asynchronously with the second test pattern.

[0106] Example 18: A non-transitory computer-readable medium comprising one or more computer-executable instructions, which, when executed by at least one processor of a computing device, cause the computing device to: display a first test pattern on a first display and display a second test pattern on a second display; acquire the displayed first test pattern and the displayed second test pattern; determine a deviation between at least one of: the acquired first test pattern and the first test pattern, or the acquired second test pattern and the second test pattern; and adjust output to at least one of the first display or the second display based on the determined deviation.

[0107] Example 19: The non-transitory computer-readable medium of Example 18, further comprising instructions for simultaneously capturing a displayed first test pattern and a displayed second test pattern in a combined image, wherein the first test pattern is likely to be different from the second test pattern.

[0108] Example 20: A non-transitory computer-readable medium according to Example 19, wherein: the instructions for determining the deviation further include instructions for distinguishing between a captured first test pattern and a captured second test pattern in a combined image; the instructions for determining the deviation further include instructions for determining a first deviation for a first display based on comparing a first set of observed features to a first set of expected features; and the instructions for adjusting the output further include instructions for applying a first correction to a first output of the first display based on the first deviation.

[0109] Embodiments of the present disclosure may include various types of artificial reality systems or be implemented in combination with various types of artificial reality systems. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user. Artificial reality may include, for example, virtual reality, augmented reality, mixed reality, hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include content that is entirely computer-generated or computer-generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional (3D) effect for a viewer). Additionally, in some embodiments, artificial reality may also be associated with an application, product, accessory, service, or some combination thereof, such as for creating content in artificial reality and / or otherwise using in artificial reality (e.g., performing an activity in artificial reality).

[0110] Artificial reality systems may be implemented in a variety of different form factors and configurations. Some artificial reality systems may be designed to operate without a near-eye display (NED). Other artificial reality systems may include a NED that also provides visibility of the real world (e.g., Figure 9 augmented reality system 900 in Figure 10 or a NED that immerses a user visually in artificial reality (e.g.,

[0111] Turning to Figure 9 , augmented reality system 900 may include a glasses device 902 having a frame 910 configured to hold a left display device 915(A) and a right display device 915(B) in front of a user's eyes. The left display device 915(A) and the right display device 915(B) may act together or independently to present an image or a series of images to the user. Although augmented reality system 900 includes two displays, embodiments of the present disclosure may be implemented in augmented reality systems having a single NED or more than two NEDs.

[0112] In some embodiments, the augmented reality system 900 may include one or more sensors, such as sensor 940. Sensor 940 may generate measurement signals in response to the movement of the augmented reality system 900 and may be generally located on any part of the frame 910. Sensor 940 may represent one or more of a variety of different sensing mechanisms, such as a position sensor, an inertial measurement unit (IMU), a depth camera assembly, a structured light emitter and / or detector, or any combination thereof. In some embodiments, the augmented reality system 900 may include or may not include sensor 940, or may include more than one sensor. In embodiments where sensor 940 includes an IMU, the IMU may generate calibration data based on the measurement signals from sensor 940. Examples of sensor 940 may include, but are not limited to, accelerometers, gyroscopes, magnetometers, other suitable types of sensors for detecting motion, sensors for error correction of the IMU, or some combination thereof.

[0113] In some examples, the augmented reality system 900 may further include a microphone array having a plurality of acoustic transducers 920(A) to 920(J), which are collectively referred to as acoustic transducers 920. The acoustic transducers 920 may represent transducers that detect changes in air pressure caused by sound waves. Each acoustic transducer 920 may be configured to detect sound and convert the detected sound into an electronic format (e.g., analog format or digital format). Figure 9 The microphone array may include, for example, ten acoustic transducers: acoustic transducers 920(A) and 920(B) that may be designed to be placed in the respective ears of a user, acoustic transducers 920(C), 920(D), 920(E), 920(F), 920(G), and 920(H) that may be positioned at various locations on the frame 910, and / or acoustic transducers 920(I) and 920(J) that may be positioned on the corresponding neckband 905.

[0114] In some embodiments, one or more of the acoustic transducers 920(A) to 920(J) may be used as output transducers (e.g., speakers). For example, acoustic transducers 920(A) and / or 920(B) may be earbuds or any other suitable type of headphones or speakers.

[0115] The construction of the acoustic transducers 920 of the microphone array may vary. Although the augmented reality system 900 is described Figure 9is shown as having ten acoustic transducers 920, but the number of acoustic transducers 920 can be more or less than ten. In some embodiments, using a greater number of acoustic transducers 920 can increase the amount of audio information collected and / or the sensitivity and accuracy of the audio information. In contrast, using a smaller number of acoustic transducers 920 can reduce the computational power required by the associated controller 950 to process the collected audio information. Additionally, the position of each acoustic transducer 920 of the microphone array can vary. For example, the position of the acoustic transducers 920 can include defined positions on the user's body, defined coordinates on the frame 910, the orientation associated with each acoustic transducer 920, or some combination thereof.

[0116] The acoustic transducers 920(A) and 920(B) can be positioned at different parts of the user's ear, such as behind the pinna, behind the tragus, and / or within the auricle or fossa. Alternatively, in addition to the acoustic transducers 920 within the ear canal, additional acoustic transducers 920 can be present on or around the ear. Positioning the acoustic transducers 920 near the user's ear canal can enable the microphone array to collect information about how sound reaches the ear canal. By positioning at least two of the plurality of acoustic transducers 920 on opposite sides of the user's head (e.g., as a binaural microphone), the augmented reality system 900 can simulate binaural hearing and capture the 3D stereo field around the user's head. In some embodiments, the acoustic transducers 920(A) and 920(B) can be connected to the augmented reality system 900 via a wired connection 930, while in other embodiments, the acoustic transducers 920(A) and 920(B) can be connected to the augmented reality system 900 via a wireless connection (e.g., a Bluetooth connection). In yet other embodiments, the acoustic transducers 920(A) and 920(B) can not be used in combination with the augmented reality system 900 at all.

[0117] The acoustic transducers 920 on the frame 910 can be positioned in a variety of different ways, including along the length of the temple, across the bridge, above or below the left display device 915(A) and the right display device 915(B), or some combination thereof. The acoustic transducers 920 can also be oriented such that the microphone array can detect sounds in a wide range of directions around the user wearing the augmented reality system 900. In some embodiments, an optimization process can be performed during the manufacture of the augmented reality system 900 to determine the relative positioning of each acoustic transducer 920 within the microphone array.

[0118] In some examples, the augmented reality system 900 may include or be connected to an external device (e.g., a paired device), such as a neckband 905. The neckband 905 generically represents any type or form of paired device. Thus, the following discussion of the neckband 905 may also apply to various other paired devices, such as a charging case, a smartwatch, a smartphone, a wristband, other wearable devices, a handheld controller, a tablet computer, a laptop computer, other external computing devices, etc.

[0119] As shown, the neckband 905 may be coupled to the eyewear device 902 via one or more connectors. These connectors may be wired or wireless, and these connectors may include electronic components and / or non-electronic components (e.g., structural components). In some cases, the eyewear device 902 and the neckband 905 may operate independently without any wired or wireless connection between them. Although Figure 9 examples of locations of components of the eyewear device 902 and the neckband 905 on the eyewear device 902 and the neckband 905 are shown, these components may be located at other locations on the eyewear device 902 and / or the neckband 905, and / or distributed in different ways on the eyewear device 902 and / or the neckband 905. In some embodiments, components of the eyewear device 902 and components of the neckband 905 may be located on one or more additional peripheral devices paired with the eyewear device 902, on the neckband 905, or some combination thereof.

[0120] Pairing an external device (e.g., the neckband 905) with the augmented reality eyewear device may enable the eyewear device to achieve the form factor of a pair of glasses while still providing sufficient battery power and computing power for the extended capabilities. Some or all of the battery power, computing resources, and / or additional features of the augmented reality system 900 may be provided by the paired device or shared between the paired device and the eyewear device, thus generally reducing the weight, heat distribution, and form factor of the eyewear device while still retaining the desired functionality. For example, the neckband 905 may allow multiple components that would otherwise be included on the eyewear device to be included in the neckband 905 because a user can tolerate a heavier weight load on their shoulders than on their head. The neckband 905 may also have a larger surface area, and using this larger surface area, heat can be diffused and dispersed into the surrounding environment. Thus, the neckband 905 may allow for a larger battery capacity and computing power compared to what might otherwise be possible on a stand-alone eyewear device. Since the weight carried in the neckband 905 is less invasive to the user than the weight carried in the eyewear device 902, a user can tolerate wearing a lighter eyewear device and carrying or wearing the paired device for a longer period of time compared to the user tolerating wearing a heavy stand-alone eyewear device, enabling the user to more fully integrate the artificial reality environment into their daily activities.

[0121] The neckband 905 can be communicatively coupled to the glasses device 902 and / or communicatively coupled to other devices. These other devices can provide certain functions for the augmented reality system 900 (e.g., tracking, positioning, depth mapping, processing, storage, etc.). In Figure 9 an embodiment, the neckband 905 can include two acoustic transducers (e.g., acoustic transducers 920(I) and 920(J)), which are part of a microphone array (or potentially form their own microphone sub-array). The neckband 905 can also include a controller 925 and a power supply 935.

[0122] The acoustic transducers 920(I) and 920(J) of the neckband 905 can be configured to detect sound and convert the detected sound into an electronic format (analog format or digital format). In Figure 9 an embodiment, the acoustic transducers 920(I) and 920(J) can be positioned on the neckband 905 to increase the distance between the acoustic transducers 920(I) and 920(J) of the neckband and other acoustic transducers 920 located on the glasses device 902. In some cases, increasing the distance between the acoustic transducers 920 of the microphone array can improve the accuracy of beamforming performed via the microphone array. For example, if sound is detected by acoustic transducers 920(C) and 920(D), and the distance between acoustic transducer 920(C) and acoustic transducer 920(D) is greater than, for example, the distance between acoustic transducer 920(D) and acoustic transducer 920(E), the determined source location of the detected sound can be more accurate than the source location determined in the case where sound is detected by acoustic transducers 920(D) and 920(E).

[0123] The controller 925 of the neckband 905 can process information generated by sensors on the neckband 905 and / or the augmented reality system 900. For example, the controller 925 can process information from a microphone array that describes the sounds detected by the microphone array. For each detected sound, the controller 925 can perform a direction-of-arrival (DOA) estimation to estimate from which direction the detected sound arrives at the microphone array. When the microphone array detects a sound, the controller 925 can populate an audio data set with the information. In embodiments where the augmented reality system 900 includes an inertial measurement unit, the controller 925 can compute all inertial and spatial operations from the IMU located on the eyewear device 902. A connector can transmit information between the augmented reality system 900 and the neckband 905, and between the augmented reality system 900 and the controller 925. This information can be in the form of optical data, electrical data, wireless data, or any other form of transmittable data. Moving the processing of the information generated by the augmented reality system 900 to the neckband 905 can reduce the weight and heat of the eyewear device 902, making the user more comfortable.

[0124] A power source 935 in the neckband 905 can provide power to the eyewear device 902 and / or the neckband 905. The power source 935 can include, but is not limited to, a lithium-ion battery, a lithium polymer battery, a disposable lithium battery, an alkaline battery, or any other form of electrical storage device. In some cases, the power source 935 can be a wired power source. Including the power source 935 on the neckband 905 rather than on the eyewear device 902 can help better distribute the weight and heat generated by the power source 935.

[0125] As described, some artificial reality systems can substantially replace one or more of a user's sensory perceptions of the real world with a virtual experience, rather than mixing artificial reality with real reality. An example of this type of system is a head-mounted display system that substantially or completely covers a user's field of view, such as Figure 10 the virtual reality system 1000 in. The virtual reality system 1000 can include a front rigid body 1002 and a band 1004 shaped to fit around a user's head. The virtual reality system 1000 can also include output audio converters 1006(A) and 1006(B). Additionally, although not shown in Figure 10 , the front rigid body 1002 can include one or more electronic components, the one or more electronic components including one or more electronic displays, one or more inertial measurement units (IMUs), one or more tracking transmitters or detectors, and / or any other suitable devices or systems for generating an artificial reality experience.

[0126] Artificial reality systems can include various types of visual feedback mechanisms. For example, the display device in the augmented reality system 900 and / or the display device in the virtual reality system 1000 can include one or more liquid crystal displays (LCDs), one or more light emitting diode (LED) displays, one or more microLED displays, one or more organic light emitting diode (OLED) displays, one or more digital light project (DLP) microdisplays, one or more liquid crystal on silicon (LCoS) microdisplays, and / or any other suitable type of display screen. These artificial reality systems can include a single display screen for both eyes, or can provide a display screen for each eye, which can provide additional flexibility for zoom adjustment or correction of the user's refractive error. Some artificial reality systems can also include optical subsystems that have one or more lenses (e.g., concave or convex lenses, Fresnel lenses, adjustable liquid lenses, etc.) through which the user can view the display screen. These optical subsystems can be used for various purposes, including collimating light (e.g., making an object appear at a distance farther than its physical distance), magnifying light (e.g., making an object appear larger than its actual size), and / or relaying light (e.g., to the viewer's eyes). These optical subsystems can be used in a non-pupil-forming architecture (e.g., a single lens configuration that directly collimates light but causes so-called pincushion distortion) and / or a pupil-forming architecture (e.g., a multi-lens configuration that produces so-called barrel distortion to eliminate pincushion distortion).

[0127] In addition to or instead of using a display screen, some of the artificial reality systems described herein may include one or more projection systems. For example, the display device in the augmented reality system 900 and / or the display device in the virtual reality system 1000 may include micro-LED projectors that project light (e.g., using waveguides) into the display device, such as a transparent combinatorial lens that allows ambient light to pass through. The display device may refract the projected light into the user's pupil and may enable the user to view both the artificial reality content and the real world simultaneously. The display device may use any of a variety of different optical components to achieve this purpose, including waveguide components (e.g., holographic waveguide elements, planar waveguide elements, diffractive waveguide elements, polarization waveguide elements, and / or reflective waveguide elements), light manipulation surfaces and elements (e.g., diffractive elements and gratings, reflective elements and gratings, and refractive elements and gratings), coupling elements, etc. The artificial reality system may also be configured to have any other suitable type or form of image projection system, such as a retinal projector for a virtual retinal display.

[0128] The artificial reality systems described herein may also include various types of computer vision components and subsystems. For example, the augmented reality system 900 and / or the virtual reality system 1000 may include one or more optical sensors, such as a two-dimensional (2D) camera or a 3D camera, a structured light emitter and detector, a time-of-flight depth sensor, a single-beam or scanning lidar sensor, a 3D lidar (LiDAR) sensor, and / or any other suitable type or form of optical sensor. The artificial reality system may process data from one or more of these sensors to identify the user's location, map the real world, provide content related to the real-world surroundings to the user, and / or perform various other functions.

[0129] The artificial reality systems described herein may also include one or more input audio converters and / or output audio converters. The output audio converters may include voice coil speakers, ribbon speakers, electrostatic speakers, piezoelectric speakers, bone conduction transducers, cartilage conduction transducers, tragus vibration transducers, and / or any other suitable type or form of audio converter. Similarly, the input audio converters may include condenser microphones, dynamic microphones, ribbon microphones, and / or any other type or form of input converter. In some embodiments, a single converter may be used for both audio input and audio output.

[0130] In some embodiments, the artificial reality systems described herein may also include tangible (i.e., haptic) feedback systems that may be incorporated into a headset, gloves, clothing, hand-held controllers, environmental devices (e.g., chairs, floor mats, etc.), and / or any other type of device or system. The haptic feedback systems may provide various types of cutaneous feedback, including vibration, force, tug, texture, and / or temperature. The haptic feedback systems may also provide various types of kinesthetic feedback, such as movement and compliance. Motors, piezoelectric actuators, fluid systems, and / or various other types of feedback mechanisms may be used to implement haptic feedback. The haptic feedback systems may be implemented independently of other artificial reality devices, within other artificial reality devices, and / or in combination with other artificial reality devices.

[0131] By providing haptic sensations, auditory content, and / or visual content, the artificial reality systems may create a complete virtual experience or enhance the user's real-world experience in a variety of scenarios and environments. For example, the artificial reality systems may assist or extend the user's perception, memory, or cognition within a particular environment. Some systems may enhance the user's interaction with others in the real world or may enable a more immersive interaction with others in a virtual world. The artificial reality systems may also be used for educational purposes (e.g., for teaching or training in schools, hospitals, government organizations, military organizations, commercial enterprises, etc.), entertainment purposes (e.g., for playing video games, listening to music, watching video content, etc.), and / or for accessibility purposes (e.g., as hearing aids, visual aids, etc.). The embodiments disclosed herein may implement or enhance the user's artificial reality experience in one or more of these scenarios and environments and / or in other scenarios and environments.

[0132] In some embodiments, the systems described herein may further include an eye tracking subsystem that is designed to identify and track various features of a user's monocular or binocular eyes (e.g., the user's gaze direction). In some examples, the term "eye tracking" may refer to a process by which the position, orientation, and / or movement of an eye are measured, detected, sensed, determined, and / or monitored. The disclosed systems may measure the position, orientation, and / or movement of an eye in a variety of different ways, including by using various optical-based eye tracking techniques, ultrasound-based eye tracking techniques, and the like. The eye tracking subsystem may be configured in a variety of different ways and may include a variety of different eye tracking hardware components or other computer vision components. For example, the eye tracking subsystem may include a variety of different optical sensors, such as a two-dimensional (2D) camera or a 3D camera, a time-of-flight depth sensor, a single-beam rangefinder or a scanning laser rangefinder, a 3D lidar sensor, and / or any other suitable type or form of optical sensor. In this example, the processing subsystem may process data from one or more of these sensors to measure, detect, determine, and / or otherwise monitor the position, orientation, and / or movement of the user's eyes.

[0133] Figure 11 is a diagram of an exemplary system 1100 that incorporates an eye tracking subsystem capable of tracking a user's monocular or binocular eyes. As Figure 11 depicted, system 1100 may include a light source 1102, an optical subsystem 1104, an eye tracking subsystem 1106, and / or a control subsystem 1108. In some examples, the light source 1102 may generate light for an image (e.g., to be presented to the viewer's eye 1101). The light source 1102 may represent any of a variety of suitable devices. For example, the light source 1102 may include a two-dimensional projector (e.g., an LCoS display), a scanning source (e.g., a scanning laser), or other devices (e.g., an LCD, an LED display, an OLED display, an active-matrix OLED (AMOLED) display, a transparent OLED (transparent OLED display, TOLED) display, a waveguide, or some other display capable of generating light for presenting an image to a viewer). In some examples, the image may represent a virtual image, which may refer to an optical image formed by the apparent divergence of light rays from a point in space, rather than an image formed by the actual divergence of light rays.

[0134] In some embodiments, the optical subsystem 1104 may receive light generated by the light source 1102 and generate converging light 1120 including an image based on the received light. In some examples, the optical subsystem 1104 may include any number of lenses (e.g., Fresnel lenses, convex lenses, concave lenses), apertures, filters, mirrors, prisms, and / or other optical components that may be combined with actuators and / or other devices. In particular, the actuator and / or other devices may translate and / or rotate one or more of the optical components to change one or more aspects of the converging light 1120. Additionally, various mechanical couplings may be used to maintain the relative spacing and / or orientation of the optical components in any suitable combination.

[0135] In one embodiment, the eye tracking subsystem 1106 may generate tracking information indicative of the gaze angle of the viewer's eye 1101. In this embodiment, the control subsystem 1108 may control aspects of the optical subsystem 1104 (e.g., the incident angle of the converging light 1120) at least in part based on the tracking information. Additionally, in some examples, the control subsystem 1108 may store and utilize historical tracking information (e.g., a history of tracking information over a given duration such as the previous second or a portion thereof) to predict the gaze angle of the eye 1101 (e.g., the angle between the visual axis of the eye 1101 and the anatomical axis). In some embodiments, the eye tracking subsystem 1106 may detect radiation emitted from a portion of the eye 1101 (e.g., the cornea, iris, or pupil, etc.) to determine the current gaze angle of the eye 1101. In other examples, the eye tracking subsystem 1106 may use a wavefront sensor to track the current position of the pupil.

[0136] Any number of techniques may be used to track the eye 1101. Some techniques may involve illuminating the eye 1101 with infrared light and measuring the reflection with at least one optical sensor tuned to be sensitive to infrared light. Information regarding how the infrared light is reflected from the eye 1101 may be analyzed to determine the one or more positions, one or more orientations, and / or one or more movements of one or more eye features (e.g., the cornea, pupil, iris, and / or retinal blood vessels).

[0137] In some examples, the radiation collected by the sensors of the eye tracking subsystem 1106 can be digitized (i.e., converted into an electronic signal). Additionally, the sensors can send a digital representation of the electronic signal to one or more processors (e.g., processors associated with a device that includes the eye tracking subsystem 1106). The eye tracking subsystem 1106 can include any of a variety of sensors in various different configurations. For example, the eye tracking subsystem 1106 can include an infrared detector that responds to infrared radiation. The infrared detector can be a thermal detector, a photon detector, and / or any other suitable type of detector. The thermal detector can include a detector that responds to the thermal effect of incident infrared radiation.

[0138] In some examples, one or more processors can process the digital representations generated by one or more sensors of the eye tracking subsystem 1106 to track the movement of the eye 1101. In another example, these processors can track the movement of the eye 1101 by executing an algorithm represented by computer-executable instructions stored on non-transitory memory. In some examples, on-chip logic (e.g., an application specific integrated circuit or ASIC) can be used to execute at least a portion of such an algorithm. As described above, the eye tracking subsystem 1106 can be programmed to use the output of one or more sensors to track the movement of the eye 1101. In some embodiments, the eye tracking subsystem 1106 can analyze the digital representations generated by the sensors to extract eye rotation information from changes in reflections. In one embodiment, the eye tracking subsystem 1106 can use the corneal reflection or the glint (also known as the Purkinje image) and / or the center of the pupil 1122 of the eye as features to be tracked over time.

[0139] In some embodiments, the eye tracking subsystem 1106 can use the center of the pupil 1122 of the eye, as well as non-collimated light in the infrared or near-infrared range, to produce a corneal reflection. In these embodiments, the eye tracking subsystem 1106 can use the vector between the center of the pupil 1122 of the eye and the corneal reflection to calculate the gaze direction of the eye 1101. In some embodiments, the disclosed system can perform a calibration process on an individual prior to tracking the user's eyes (e.g., using supervised or unsupervised techniques). For example, the calibration process can include: guiding the user to look at one or more points displayed on a display while the eye tracking system records values corresponding to each fixation location (which is associated with each point).

[0140] In some embodiments, the eye tracking subsystem 1106 can use two types of infrared and / or near-infrared (also known as active light) eye tracking techniques: bright pupil eye tracking and dark pupil eye tracking, which can be distinguished based on the position of the illumination source relative to the optical elements used. If the illumination is coaxial with the optical path, the eye 1101 can act as a retroreflector because light is reflected back from the retina, resulting in a bright pupil effect similar to the red-eye effect in photography. If the illumination source is offset from the optical path, the pupil 1122 of the eye may appear darker because the retroreflection from the retina is directed away from the sensor. In some embodiments, bright pupil tracking can produce a greater iris / pupil contrast, allowing for more robust eye tracking with iris pigmentation and reducing interference (e.g., interference caused by eyelashes and other blurring features). Bright pupil tracking can also allow tracking under illumination conditions ranging from completely dark to very bright environments.

[0141] In some embodiments, the control subsystem 1108 can control the light source 1102 and / or the optical subsystem 1104 to reduce optical aberrations (e.g., chromatic aberration and / or monochromatic aberration) of the image that may be caused by or affected by the eye 1101. In some examples, as described above, the control subsystem 1108 can use the tracking information from the eye tracking subsystem 1106 to perform this control. For example, when controlling the light source 1102, the control subsystem 1108 can change the light generated by the light source 1102 (e.g., through image rendering) to modify the image (e.g., pre-distort) and thus reduce the aberration of the image caused by the eye 1101.

[0142] The disclosed system can track both the position and relative size of the pupil (e.g., due to pupil dilation and / or constriction). In some examples, for different types of eyes, the eye tracking devices and components (e.g., sensors and / or sources) used to detect and / or track the pupil can be different (or calibrated differently). For example, for eyes of different colors, and / or different pupil types and / or sizes, etc., the frequency range of the sensor can be different (or calibrated individually). Thus, the various eye tracking components described herein (e.g., infrared sources and / or sensors) may need to be calibrated for each individual user and / or eye.

[0143] The disclosed system can track eyes with and without ophthalmic correction (e.g., correction provided by contact lenses worn by the user). In some embodiments, the ophthalmic correction element (e.g., an adjustable lens) can be directly incorporated into the artificial reality system described herein. In some examples, the color of the user's eyes may require modification of the corresponding eye tracking algorithm. For example, the eye tracking algorithm may need to be modified at least in part based on the different color contrasts between brown eyes and, for example, blue eyes.

[0144] Figure 12 Yes Figure 11 Figure 11 is a more detailed illustration of various aspects of the eye tracking subsystem shown. As shown in this figure, the eye tracking subsystem 1200 may include at least one source 1204 and at least one sensor 1206. The source 1204 generally represents any type or form of element capable of emitting radiation. In one example, the source 1204 may generate visible radiation, infrared radiation, and / or near-infrared radiation. In some examples, the source 1204 may radiate the non-collimated infrared portion and / or near-infrared portion of the electromagnetic spectrum towards the user's eye 1202. The source 1204 may utilize various sampling rates and sampling speeds. For example, the disclosed system may use a source with a higher sampling rate in order to capture the fixation eye movements of the user's eye 1202 and / or correctly measure the saccade dynamics of the user's eye 1202. As described above, any type or form of eye tracking technology may be used to track the user's eye 1202, and these types or forms of eye tracking technology include optical-based eye tracking technology, ultrasound-based eye tracking technology, and the like.

[0145] The sensor 1206 generally represents any type or form of element capable of detecting radiation, such as radiation reflected from the user's eye 1202. Examples of the sensor 1206 include, but are not limited to, charge-coupled device (CCD), photodiode array, and / or complementary metal-oxide-semiconductor (CMOS)-based sensor devices, and the like. In one example, the sensor 1206 may represent a sensor with predetermined parameters, and these parameters include, but are not limited to, dynamic resolution range, linearity, and / or other characteristics specifically selected and / or designed for eye tracking.

[0146] As described above, the eye tracking subsystem 1200 may generate one or more glints. As described above, the glint 1203 may represent the reflection of radiation (e.g., infrared radiation from an infrared source (such as the source 1204)) from the structure of the user's eye. In various embodiments, an eye tracking algorithm executed by a processor (inside or outside the artificial reality device) may be used to track the glint 1203 and / or the user's pupil. For example, the artificial reality device may include: a processor and / or a storage device for locally executing eye tracking; and / or a transceiver for sending and receiving data required for executing eye tracking on an external device (such as a mobile phone, a cloud server, or other computing devices).

[0147] Figure 12Shows an example image 1205 collected by an eye tracking subsystem (e.g., eye tracking subsystem 1200). In this example, image 1205 can include both the user's pupil 1208 and a glint 1210 near the user's pupil. In some examples, an artificial intelligence-based algorithm (e.g., a computer vision-based algorithm) can be used to identify the pupil 1208 and / or the glint 1210. In one embodiment, image 1205 can represent a single frame in a series of frames, and the series of frames can be continuously analyzed to track the user's eyes 1202. Additionally, the pupil 1208 and / or the glint 1210 can be tracked over a period of time to determine the user's gaze.

[0148] In one example, the eye tracking subsystem 1200 can be configured to identify and measure the user's inter-pupillary distance (IPD). In some embodiments, the eye tracking subsystem 1200 can measure and / or calculate the user's IPD when the user is wearing an artificial reality system. In these embodiments, the eye tracking subsystem 1200 can detect the position of the user's eyes and can use this information to calculate the user's IPD.

[0149] As mentioned, the eye tracking system or eye tracking subsystem disclosed herein can track the user's eye position and / or eye movement in various ways. In one example, one or more light sources and / or optical sensors can collect images of the user's eyes. Then, the eye tracking subsystem can use the collected information to determine the user's inter-pupillary distance, inter-ocular distance, and / or the 3D position of each eye (e.g., for distortion adjustment purposes), the collected information including the magnitude of the torsion and rotation of each eye (i.e., roll, up and down movement, and left and right movement) and / or the gaze direction. In one example, infrared light can be emitted by the eye tracking subsystem and reflected from each eye. The reflected light can be received or detected by an optical sensor, and the reflected light can be analyzed to extract eye rotation data from the changes in the infrared light reflected from each eye.

[0150] The eye tracking subsystem can use any of a variety of different methods to track a user's eyes. For example, a light source (e.g., an infrared light-emitting diode) can project a dot pattern onto each of the user's eyes. The eye tracking subsystem can then detect (e.g., via an optical sensor coupled to the artificial reality system) the reflection of the dot pattern from each of the user's eyes and analyze the reflection to identify the position of each of the user's pupils. Thus, the eye tracking subsystem can track up to six degrees of freedom for each eye (i.e., 3D position, roll, up and down movement, and left and right movement), and can combine at least one subset of the tracked quantities from the user's two eyes to estimate the fixation point (i.e., the 3D position or the position in the virtual scene that the user is viewing) and / or IPD.

[0151] In some cases, when the user's eyes move in different directions to view, the distance between the user's pupils and the display can change. When the viewing direction changes, the changing distance between the pupils and the display can be referred to as "pupil wander", and when the distance between the pupils and the display changes, it may cause distortion perceived by the user due to the light focusing at different positions. Thus, measuring the distortion at different eye positions and pupil distances relative to the display and generating distortion corrections for different positions and distances can allow the distortion caused by pupil wander to be mitigated by tracking the 3D position of the user's eyes and applying the distortion correction corresponding to the 3D position of each of the user's eyes at a given point in time. Thus, knowing the 3D position of each of the user's eyes can allow the distortion caused by the change in the distance between the pupils of the eyes and the display to be mitigated by applying the distortion correction to each 3D eye position. Additionally, as mentioned above, knowing the position of each of the user's eyes can also enable the eye tracking subsystem to automatically adjust the user's IPD.

[0152] In some embodiments, the display subsystem can include various additional subsystems that can work in conjunction with the eye tracking subsystem described herein. For example, the display subsystem can include a zoom subsystem, a scene rendering module, and / or a vergence processing module. The zoom subsystem can cause the left and right display elements to change the focal length of the display device. In one embodiment, the zoom subsystem can physically change the distance between the display and the optics by moving the display, the optics, or both, through which the display can be viewed. Additionally, moving two lenses relative to each other or translating them can also be used to change the focal length of the display. Thus, the zoom subsystem can include an actuator or motor that moves the display and / or the optics to change the distance between them. The zoom subsystem can be separate from the display subsystem or integrated into the display subsystem. The zoom subsystem can also be integrated into its actuator subsystem and / or the eye tracking subsystem described herein, or separate from its actuator subsystem and / or the eye tracking subsystem described herein.

[0153] In one example, the display subsystem may include a vergence processing module configured to determine the vergence depth of a user's gaze based on the fixation point determined by the eye tracking subsystem and / or an estimated intersection point of the gaze lines. Vergence may refer to the simultaneous movement or rotation of both eyes in opposite directions to maintain binocular single vision, which can be performed naturally and automatically by the human eye. Thus, the position towards which the user's eyes tend is the position the user is looking at and is typically also the position where the user's eyes are focused. For example, the vergence processing module may triangulate the gaze lines to estimate the distance or depth from the user associated with the intersection point of the gaze lines. Then, the depth associated with the intersection point of the gaze lines can be used as an approximation of the focusing distance, which can identify the distance from the user towards which the user's eyes are directed. Thus, the vergence distance can allow determination of the position at which the user's eyes should be focused and the depth from the user's eyes at which the user's eyes are focused, thereby providing information for rendering adjustments of a virtual scene (e.g., an object or a focal plane).

[0154] The vergence processing module may cooperate with the eye tracking subsystem described herein to adjust the display subsystem to account for the user's vergence depth. When the user is focused on something far away, the user's pupils may be slightly more separated than when the user is focused on something close. The eye tracking subsystem may obtain information about the user's vergence or focusing depth and may adjust the display subsystem to be closer when the user's eyes are focused or converged on something close and to be farther away when the user's eyes are focused or converged on something far away.

[0155] The eye tracking information generated by the eye tracking subsystem described above may also be used, for example, to modify various aspects of how different computer-generated images are presented. For example, the display subsystem may be configured to modify at least one aspect of how a computer-generated image is presented based on the information generated by the eye tracking subsystem. For example, a computer-generated image may be modified based on the user's eye movements such that if the user is looking up, the computer-generated image may be moved up on the screen. Similarly, if the user is looking to one side or down, the computer-generated image may be moved to one side or down on the screen. If the user closes their eyes, the computer-generated image may be paused or removed from the display and, once the user opens their eyes again, the computer-generated image will resume.

[0156] The above-described eye-tracking subsystem can be incorporated into one or more of the various artificial reality systems described herein in a variety of ways. For example, one or more components of system 1100 and / or the eye-tracking subsystem 1200 can be incorporated into Figure 9 the augmented reality system 900 and / or Figure 10 the virtual reality system 1000, enabling these systems to perform various eye-tracking tasks (including one or more of the eye-tracking operations described herein).

[0157] As described above, the computing devices and systems described and / or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those included in the modules described herein. In their most basic configuration, these computing devices can each include at least one storage device and at least one physical processor.

[0158] In some examples, the term "storage device" generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, the storage device can store, load, and / or maintain one or more of the modules described herein. Examples of storage devices include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid state drive (SSD), optical disk drive, cache, variations or combinations of one or more of the above, or any other suitable storage memory.

[0159] In some examples, the term "physical processor" generally refers to any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, the physical processor can access and / or modify one or more of the modules stored in the above storage device. Examples of physical processors include, but are not limited to, microprocessors, microcontrollers, central processing unit (CPU), field programmable gate array (FPGA) implementing a soft-core processor, application specific integrated circuit (ASIC), portions of one or more of the above, variations or combinations of one or more of the above, or any other suitable physical processor.

[0160] Although described and / or shown as separate elements, the modules described and / or shown herein may represent a single module or portions of an application. Additionally, in some embodiments, one or more of these modules may represent one or more software applications or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks. For example, one or more of the modules described and / or shown herein may represent modules stored on and configured to run on one or more of the computing devices or systems described and / or shown herein. One or more of these modules may also represent all or portions of one or more special-purpose computers configured to perform one or more tasks.

[0161] Additionally, one or more of the modules described herein may transform data, physical devices, and / or representations of physical devices from one form to another. For example, one or more of the modules described herein may receive image data to be transformed, transform the image data, output the result of the transformation to determine a deviation, use the result of the transformation to determine a correction, and store the result of the transformation to correct an output frame. Additionally or alternatively, one or more of the modules described herein may transform a processor, volatile memory, non-volatile memory, and / or any other portion of a physical computing device from one form to another by executing on the computing device, storing data on the computing device, and / or otherwise interacting with the computing device.

[0162] In some embodiments, the term “computer-readable medium” generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transmissive media and non-transitory media, such as a carrier wave for the transmissive media and magnetic storage media (e.g., hard disk drives, tape drives, and floppy disks), optical storage media (e.g., Compact Disk (CD), Digital Video Disk (DVD), and Blu-ray Discs), electronic storage media (e.g., solid state drives and flash media), and other distribution systems for the non-transitory media.

[0163] The process parameters and step sequences described and / or shown herein are given only as examples and may be varied as needed. For example, although multiple steps shown and / or described herein may be shown or discussed in a particular order, those steps need not necessarily be performed in the order shown or discussed. The various exemplary methods described and / or shown herein may also omit one or more of the steps described or shown herein, or may include additional steps other than those disclosed.

[0164] The foregoing description is provided to enable other technicians in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or limited to any precise form. Many modifications and variations are possible without departing from the spirit and scope of the disclosure. The embodiments disclosed herein should be considered illustrative rather than restrictive in all respects. In determining the scope of the disclosure, reference should be made to any appended claims and their equivalents.

[0165] Unless otherwise stated, as used in this specification and the claims, the terms "connected to" and "coupled to" (and their derivatives) will be interpreted to allow both direct and indirect connection (i.e., indirect connection via other elements or components). In addition, as used in the specification and the claims, the term "a" or "an" will be interpreted to mean "at least one of...". Finally, for ease of use, as used in the specification and the claims, the terms "comprising" and "having" (and their derivatives) may be interchanged with the word "including" and have the same meaning as the word "including".

Claims

1. A computer-implemented method comprising: displaying a first test pattern on a first display and displaying a second test pattern on a second display; collecting a displayed first test pattern and a displayed second test pattern; determining a deviation between at least one of: a first acquired test pattern and the first test pattern, or a second acquired test pattern and the second test pattern; as well as Based on the determined deviation, an output to at least one of the first display or the second display is adjusted.

2. The method according to claim 1, further comprising: The displayed first test pattern and the displayed second test pattern are simultaneously captured in a combined image, wherein the first test pattern is highly likely to be different from the second test pattern.

3. The method according to claim 2, wherein: Determining the deviation further includes distinguishing the acquired first test pattern from the acquired second test pattern in the combined image.

4. The method according to claim 3, wherein: Distinguishing the acquired first test pattern from the acquired second test pattern further comprises: identifying a first set of expected features from the first test pattern and a second set of expected features from the second test pattern; identifying a plurality of observed features from the combined image; and A first set of observed features from the plurality of observed features is determined to correspond to the first test pattern based on similarity to the first set of expected features, and a second set of observed features from the plurality of observed features is determined to correspond to the second test pattern based on similarity to the second set of expected features.

5. The method according to claim 4, wherein: Determining the deviation further comprises: determining a first deviation for the first display based on comparing the first set of observed features to the first set of expected features; and Adjusting the output further includes applying a first correction to a first output of the first display based on the first deviation.

6. The method according to claim 5, wherein: Identifying the first set of expected features includes: constructing a dot model based on feature locations from the first test pattern; Identifying the plurality of observed features comprises: determining locations of identified features from the combined image; Distinguishing the acquired first test pattern from the acquired second test pattern further comprises: selecting the first set of observed features from the plurality of observed features based on a point relationship matrix between the point model and the first set of observed features; and Determining the first deviation is based on the point relationship matrix.

7. The method according to claim 6, wherein: The first correction is based on the point relationship matrix.

8. The method according to claim 1, further comprising: When a blink of the user is detected, at least one of the first test pattern or the second test pattern is displayed.

9. The method according to claim 1, further comprising: When it is detected that the user is looking away from at least one of the first display or the second display, at least one of the first test pattern or the second test pattern is displayed.

10. The method according to claim 1, further comprising: At least one of the first test pattern or the second test pattern hidden in an output frame is displayed.

11. The method according to claim 1, further comprising: The first test pattern is displayed asynchronously with the second test pattern.

12. A system comprising: at least one physical processor; a first display; Second display; Image sensor devices; as well as a physical memory, the physical memory comprising computer executable instructions, which when executed by the physical processor, cause the physical processor to: displaying a first test pattern on the first display and displaying a second test pattern on the second display; using the image sensor device to capture a displayed first test pattern and a displayed second test pattern; determining a deviation between at least one of: a first acquired test pattern and the first test pattern, or a second acquired test pattern and the second test pattern; as well as Based on the determined deviation, an output to at least one of the first display or the second display is adjusted.

13. The system according to claim 12, wherein: The image sensor device is configured to simultaneously capture the displayed first test pattern and the displayed second test pattern in a combined image, wherein the first test pattern is likely to be different from the second test pattern.

14. The system according to claim 13, wherein: The instructions for determining the deviation further include instructions for distinguishing the acquired first test pattern from the acquired second test pattern in the combined image by: identifying a first set of expected features from the first test pattern and a second set of expected features from the second test pattern; identifying a plurality of observed features from the combined image; as well as A first set of observed features from the plurality of observed features is determined to correspond to the first test pattern based on similarity to the first set of expected features, and a second set of observed features from the plurality of observed features is determined to correspond to the second test pattern based on similarity to the second set of expected features.

15. The system of claim 14, wherein: The instructions for determining the deviation further include: instructions for determining a first deviation for the first display based on comparing the first set of observed features to the first set of expected features; and The instructions for adjusting the output further include instructions for applying a first correction to a first output of the first display based on the first deviation.

16. The system of claim 15, wherein: The instructions for identifying the first set of expected features include: instructions for constructing a dot model based on feature locations from the first test pattern; The instructions for identifying the plurality of observed features include: instructions for determining locations of the identified features from the combined image; The instructions for distinguishing the acquired first test pattern from the acquired second test pattern further include: instructions for selecting the first set of observed features from the plurality of observed features based on a point relationship matrix between the point model and the first set of observed features; Determining the first deviation is based on the point relationship matrix; and The first correction is based on the point relationship matrix.

17. The system of claim 12, wherein: The instructions for further displaying the first test pattern and the second test pattern include at least one of the following: instructions for displaying at least one of the first test pattern or the second test pattern when a user blink is detected; instructions for displaying at least one of the first test pattern or the second test pattern upon detecting that a user is looking away from at least one of the first display or the second display; instructions for displaying at least one of the first test pattern or the second test pattern hidden in an output frame; as well as Instructions for causing the first test pattern to be displayed asynchronously with the second test pattern.

18. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to: displaying a first test pattern on a first display and displaying a second test pattern on a second display; collecting a displayed first test pattern and a displayed second test pattern; determining a deviation between at least one of: a first acquired test pattern and the first test pattern, or a second acquired test pattern and the second test pattern; as well as Based on the determined deviation, an output to at least one of the first display or the second display is adjusted.

19. The non-transitory computer readable medium of claim 18, further comprising instructions for simultaneously acquiring the displayed first test pattern and the displayed second test pattern in a combined image, wherein: The first test pattern is highly likely to be different from the second test pattern.

20. The non-transitory computer readable medium of claim 19, wherein: The instructions for determining the deviation further include: instructions for distinguishing, in the combined image, the acquired first test pattern and the acquired second test pattern; The instructions for determining the deviation further include: instructions for determining a first deviation for the first display based on comparing the first set of observed features to the first set of expected features; and The instructions for adjusting the output further include instructions for applying a first correction to a first output of the first display based on the first deviation.

Images