Packet gate scanning techniques under foveated displays

Abstract

A packet gate scanning technique under a foveated display is provided. A display system is described that includes a plurality of gate drive units, each gate drive unit coupled to a subset of a plurality of clock signal lines. The clock signal selectively turned on or off during a given time period may cause one or more of the plurality of gate driving units to generate one or more gate signals, each driving a row of display elements in the display. The clock signal may be controlled and may be coupled to the gate driving unit such that a gate signal may drive a different number of display element rows based on a combined state of the clock signal. The display system may also include a plurality of demultiplexers configured to simultaneously relay the display data to one or more columns of display elements of the display.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 665,049, filed June 27, 2024, entitled “Grouped Gate Scanning, Grouped Demultiplexing and Macro-Pixel Generation in LCDs,” and U.S. Non-Provisional Patent Application No. 19 / 242,340, filed June 18, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to techniques for driving foveated displays. Background Technology

[0004] Foveated imaging is a display technique in which the image resolution on a single image can be varied. For example, the portion of the image corresponding to the center of the retina (fovea) can be rendered at a higher resolution than the portion of the image farther from the retina. Because the peripheral areas of the eye have lower contrast sensitivity compared to their center, the image area rendered at a lower resolution may not be noticeable. Summary of the Invention

[0005] Various embodiments of this disclosure can provide a display system including a plurality of gate driving units, each gate driving unit coupled to a subset of a plurality of clock signal lines. A clock signal selectively turned on or off during a given time period can cause one or more of the plurality of gate driving units to generate one or more gate signals, each gate signal driving a row of display elements in the display. The clock signals can be controlled and coupled to the gate driving units such that, based on a combination of clock signal states, a gate signal can drive a different number of rows of display elements. For example, the clock signals can be generated in a manner that causes only a single gate driving unit to drive only a single row of display elements using a single gate signal, and the clock signals can be generated in another manner that causes one or more gate driving units to simultaneously drive multiple rows of display elements. The clock signals can be provided to the plurality of gate driving units to drive each row of display elements, wherein each row is driven independently, or as part of a group of display elements consisting of two rows, three rows, four rows, etc. This process can allow multiple display elements to be driven simultaneously, thereby potentially reducing the amount of time required to drive all display elements in the display and increasing the frame rate. As mentioned above, in some parts of the image displayed in the fovea, areas may be displayed at a lower resolution, and this process can allow for the production of such images while potentially increasing the frame rate. Attached Figure Description

[0006] The accompanying drawings illustrate several exemplary embodiments and are part of this specification. These drawings, together with the following description, illustrate and explain various principles of this disclosure.

[0007] Figure 1 An illustrative concave display according to some embodiments of the present disclosure is depicted.

[0008] Figure 2 Addressing of display elements in a display according to some embodiments of the present disclosure is described.

[0009] Figure 3 Methods for driving rows of display elements in a display according to some embodiments of the present disclosure are described.

[0010] Figure 4 A portion of an illustrative display system according to some embodiments of the present disclosure is depicted, the system being configured to adjust the number of rows of display elements driven simultaneously.

[0011] Figure 5 Some embodiments according to this disclosure are depicted. Figure 4 An illustrative embodiment of the gate driving unit shown.

[0012] Figures 6 to 10Examples of operating a display system to drive rows of display elements according to some embodiments of the present disclosure are depicted.

[0013] Figure 11 Examples of driving display element columns according to some embodiments of the present disclosure are depicted.

[0014] Figure 12 A method for simultaneously driving two sub-pixel columns according to some embodiments of the present disclosure is described.

[0015] Figure 13 Examples of 1:2 demultiplexers according to some embodiments of this disclosure are described.

[0016] Figure 14 Examples of 1:3 demultiplexers according to some embodiments of this disclosure are described.

[0017] Figure 15 An illustrative system is described, which can be operated according to the techniques described herein, according to some embodiments of the present disclosure.

[0018] Figure 16 This is an illustration of an example artificial reality system according to some embodiments of the present disclosure.

[0019] Figure 17 This is an illustration of an example artificial reality system with a handheld device according to some embodiments of the present disclosure.

[0020] Figure 18A These are illustrations of example user interactions within an artificial reality system according to some embodiments of the present disclosure.

[0021] Figure 18B These are illustrations of example user interactions within an artificial reality system according to some embodiments of the present disclosure.

[0022] Figure 19A These are illustrations of example user interactions within an artificial reality system according to some embodiments of the present disclosure.

[0023] Figure 19B These are illustrations of example user interactions within an artificial reality system according to some embodiments of the present disclosure.

[0024] Figure 20 This is an illustration of an example wrist-worn wearable device of an artificial reality system according to some embodiments of the present disclosure.

[0025] Figure 21 This is an illustration of an example wearable artificial reality system according to some embodiments of the present disclosure.

[0026] Figure 22 This is an illustration of an example augmented reality system according to some embodiments of the present disclosure.

[0027] Figure 23A This is an illustration of an example virtual reality system according to some embodiments of the present disclosure.

[0028] Figure 23B yes Figure 23A The illustration shows another perspective of the virtual reality system.

[0029] Figure 24 It is a block diagram showing the system components of an example artificial reality system and an example virtual reality system.

[0030] Figure 25 This is a diagram of an example system that includes an eye-tracking subsystem capable of tracking one or both of a user's eyes.

[0031] Figure 26 yes Figure 25 A more detailed illustration of the various aspects of the eye-tracking subsystem shown.

[0032] Throughout the accompanying drawings, the same reference numerals and descriptions denote similar but not necessarily identical elements. While the exemplary embodiments described herein are readily adaptable to various modifications and alternatives, several specific embodiments are illustrated by way of example in these drawings, and these specific embodiments will be described in detail herein. However, the exemplary embodiments described herein are not intended to be limited to the specific forms disclosed. Rather, this disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims. Detailed Implementation

[0033] This disclosure generally relates to techniques for driving foveated displays. As will be explained in more detail below, embodiments of this disclosure may include a controller that generates drive signals to address display elements in the foveated display, wherein the display elements can be addressed individually or in groups to produce a foveated display image. Some displays can be driven by sequentially activating consecutive individual display elements in the display. For example, selected rows and selected columns can be driven simultaneously to address a single display element that is in both the selected row and the selected column. However, this approach can limit the production of very high-resolution images due to high power consumption and / or because the available computing power is too high to produce the desired frame rate. This is especially true in portable devices such as wearable devices (e.g., artificial reality devices), which are particularly limited by power capabilities and / or computing power.

[0034] Various embodiments of this disclosure can provide a display system including a plurality of gate driving units, each gate driving unit coupled to a subset of a plurality of clock signal lines. A clock signal selectively turned on or off during a given time period can cause one or more of the plurality of gate driving units to generate one or more gate signals, each gate signal driving a row of display elements in the display. The clock signals can be controlled and coupled to the gate driving units such that, based on a combination of clock signal states, a gate signal can drive a different number of rows of display elements. For example, the clock signals can be generated in a manner that causes only a single gate driving unit to drive only a single row of display elements using a single gate signal, and the clock signals can be generated in another manner that causes one or more gate driving units to simultaneously drive multiple rows of display elements. The clock signals can be provided to the plurality of gate driving units to drive each row of display elements, wherein each row is driven independently, or as part of a group of display elements consisting of two rows, three rows, four rows, etc. This process can allow multiple display elements to be driven simultaneously, thereby potentially reducing the amount of time required to drive all display elements in the display and increasing the frame rate. As mentioned above, in some parts of the image displayed in the fovea, areas may be displayed at a lower resolution, and this process can allow for the production of such images while potentially increasing the frame rate.

[0035] Various embodiments of this disclosure can provide a display system including a plurality of demultiplexers configured to simultaneously provide display data to one or more columns of display elements of a display, wherein the plurality of demultiplexers includes demultiplexers configured to simultaneously provide display data to different numbers of columns. For example, a first demultiplexer may be configured to simultaneously provide display data to two different columns of display elements, while a second demultiplexer may be configured to simultaneously provide display data to four different columns of display elements. The display data may, for example, be analog signals for controlling the color and / or brightness of display elements (e.g., analog signals controlling the brightness of subpixels of a certain color). A controller may select a desired demultiplexer from the plurality of demultiplexers to receive a portion of the display data based on image data describing a centrally located image of the display. The same display data can be provided to multiple columns of display elements simultaneously, thereby allowing multiple columns to be driven using the same image data. This process can allow multiple columns of display elements to be driven simultaneously to produce the same color, thereby potentially reducing the amount of data required to drive all display elements in the display.

[0036] Various embodiments of this disclosure can provide a display system in which both of the above methods are implemented simultaneously, thereby allowing multiple rows and columns of display elements to be driven simultaneously, such that a group of display units operate in the same manner, potentially reducing data volume and increasing frame rate.

[0037] As a further explanation, Figure 1 An illustrative concave display according to some embodiments of the present disclosure is depicted, the concave display comprising a grid of 576 (24 × 24) display elements (e.g., pixels and / or subpixels). Figure 1 In the example, an image was generated using a concave pattern that includes a central high-detail area, where each of the 64 display elements in that area independently produces light of the desired color and / or brightness. In this concave pattern, a medium-detail area surrounds the high-detail area, and a low-detail area surrounds the medium-detail area. In the medium-detail area, four groups of display elements (2×2 blocks) are operated to produce the same output. In principle, when a display generates an image using the depicted concave pattern, any display element in the high-detail area might appear different from each other, while in the medium-detail area, each 2×2 block will appear consistent, effectively acting as a larger display element. Similarly, in... Figure 1 In the low detail region, a 4×4 block of display elements is manipulated to produce the same output, thus effectively acting as a single display element larger than the display elements in the medium detail region. A group of multiple display elements operating to produce the same output may be referred to herein as a macropixel.

[0038] It will be recognized that, generally speaking, high-detail areas can be geographically distributed across different locations on the display depending on the image frame, and Figure 1 This is merely an illustrative example of a centrally recessed pattern, where the high-detail area is precisely located in the center. Furthermore, any number of consecutive display elements arranged in any suitable size and shape can be manipulated to produce the same output in any given area. For example, a 3×3 block or a 5×5 block of display elements can be manipulated in this way in a given area of ​​an image. Similarly, any number of areas with different levels of detail can be generated in a given image, and the techniques described herein are not limited to this. Figure 1 The example shows three regions.

[0039] exist Figure 1In the example, the display element is addressed when both gate signal 102 and display signal 101 are provided to it. As shown, the gate signal can be provided along the rows of the display, and the display signal along the columns. Thus, a single display element can be addressed by providing a gate signal to the row where the display element is located and a display signal to the column where the display element is located. According to some embodiments, each gate signal can be a digital value (e.g., 0 or 1) indicating whether the row is active, and each display signal can be an analog signal indicating how to operate the display element (e.g., indicating a brightness level). In a typical display, an image is generated by sequentially activating rows of the display via successive gate signals, and as each gate signal activates a given row, a display signal is provided along successive columns to indicate how to operate each display element. In this way, each display element is operated to generate light (e.g., from top left to bottom right, element by element, row by row).

[0040] Figure 2 This type of addressing according to some embodiments of this disclosure is described in more detail. Figure 2 The example depicts six display elements arranged in three distinct rows and two distinct columns, which could represent a portion of a larger array of display elements with many rows and columns. Figure 2 As shown, each of the display elements 211, 212, 221, 222, 231, and 232 can be activated by providing both display signal inputs and gate signal inputs along both columns and rows, respectively. For example, display element 211 is activated when both gate signal 1 and display signal 1 are active; otherwise, display element 211 is not activated. As described above, typically, display elements in a display are activated sequentially, one by one, to generate a single image frame. For example, in... Figure 2 In the example, gate signal 1 can be set to active and display signal 1 is provided, causing display element 211 to operate according to display signal 1; then, display signal 1 can be stopped and display signal 2 can be provided, while gate signal 1 remains active, causing display element 212 to operate according to display signal 2; and so on.

[0041] Some display drivers coordinate this row-by-row addressing process by generating a continuous gate signal for each row. In some arrangements, the gate signal is generated by circuitry that sends a valid signal to the desired row by receiving a control input (e.g., a clock signal) associated with that row.

[0042] Figure 3 A method for driving rows of display elements in a display, according to some embodiments of this disclosure, is described. For comparative purposes, Figure 3The gate signals described herein are generated during the process of driving each row of display elements independently (one display element at a time). Such a process can be used to generate portions of the recessed display image (e.g., when generating...). Figure 1 (When showing high detail areas), although as mentioned above, some display systems always operate in this way.

[0043] exist Figure 3 In the example, in each display cycle ( Figure 3 During the period indicated by “H” in the middle and subsequent figures, one of the gate signals in each gate signal is driven high to, as shown in the figure below. Figure 2 The associated display element row is activated as shown. In this example, there are only six display element rows, but the same method can be performed on any number of rows. Figure 3 In the example, two images can be generated consecutively in this illustrative miniature display by sequentially driving each row to generate the first image, then sequentially driving each row again to generate the second image, and so on. During each display cycle, each column of display elements can be driven one by one, such that during each display cycle, the desired color and / or brightness of each display element in a row can be controlled.

[0044] Figure 4 A portion of an illustrative display system according to some embodiments of the present disclosure is depicted, the system being configured to adjust the number of rows of display elements driven simultaneously, thereby driving all rows more efficiently to produce a centrally recessed display image. Figure 4 In the example, multiple gate drive units 410 are each coupled to some (but not all) of the eight clock signal lines labeled C1 to C8. For example, one gate drive unit 410 (via clock inputs CKA, CKB, CKC, CKD, CKE, and CKF, respectively) is coupled to clock signal lines C1, C2, C3, C4, C6, and C7, while another gate drive unit 410 (via clock inputs CKA, CKB, CKC, CKD, CKE, and CKF, respectively) is coupled to clock signal lines C2, C3, C5, C6, C7, and C8. The upper gate drive units generate gate signal outputs 421, 422, 423, and 424, while the lower gate drive units generate gate signal outputs 425, 426, 427, and 428.

[0045] As will be further described below, by controlling each of the signals on clock signal lines C1 to C8, each of the eight gate signals 1 to 8 can be generated simultaneously in a combination of at least one or two gate signals. In principle, this circuit can be used to generate combinations of three, four or more gate signals generated simultaneously, or to generate these combinations using circuitry that includes more than eight gate signals, and in some cases, the gate drive unit has additional inputs in addition to those in the example of gate drive unit 410.

[0046] exist Figure 4 In the example, any number of instances of gate driving units 410 may be included, and each pair of gate driving units may be coupled to a clock signal line shown in the display device, such that any number of display element rows can be driven by the corresponding gate signal. Figure 4 In the example, each gate drive unit 410 in the gate drive unit chain (two gate drive units in the gate drive unit chain are shown in the figure) enables the continuous gate drive via the SETU input of each driver during top-down scanning, or via the SETD input of each driver during bottom-up scanning.

[0047] Figure 5 Some embodiments according to this disclosure are depicted. Figure 4 An illustrative embodiment of the gate driving unit 410 shown. Figure 5 In the example, when the corresponding clock input is high, and when both the CKE and CKF clock inputs are low, the respective gate signals 1 through 4 are generated (e.g., gate signal 1 is generated when the input to CKA is high). Figure 5 In the example, INITB is the initial reset signal, which sets all gate signals 1 to 4 to low; UD and UDB are signals that can be set to indicate whether the scan from gate drive unit to gate drive unit is from top to bottom or from bottom to top; SETU is the activation signal for the top-down scan; and SETD is the activation signal for the bottom-up scan.

[0048] Back Figure 4 It should be noted that the clock inputs CKB and CKC of the lower gate drive unit 410 are connected to clock signal lines C6 and C7, and clock signal lines C6 and C7 are also connected to the clock inputs CKE and CKF of the upper gate drive unit 410. Therefore, when any consecutive clock input pair to the lower gate drive unit (e.g., CKA and CKB, or CKB and CKC, etc.) is activated, this will suppress any gate signal output from the upper gate drive unit. Figure 6 , Figure 7 , Figure 8 , Figure 9 and Figure 10 As explained, this configuration provides various desired combinations of simultaneously generating one gate signal, two gate signals, three gate signals, four gate signals, etc.

[0049] exist Figure 6 In the example, within each display cycle (again denoted as "H"), one of the clock signals C1 through C8 is activated, where after the last clock signal C8 goes high, the first clock signal C1 is driven high again, and so on. This can be achieved by arranging the clock signals as follows: Figure 4 The chain of gate driving units 410 shown allows a gate signal to be applied to one row of display elements during each display cycle. For example, when C1 is high, the clock input CKA of the first gate driving unit is high, resulting in a gate signal output at 421. When C6 is high, the clock input CKE of the upper gate driving unit is high, and the clock input CKB of the lower gate driving unit is high. However, since a high-level input to input CKE suppresses the upper gate driving unit from generating a gate signal output, only the lower gate driving unit (at 426) generates an output gate signal.

[0050] Figure 6 behavior and Figure 3 The behavior shown is the same, but importantly, it can be controlled row by row to generate multiple gate signals simultaneously as needed. For example, when controlling the display elements to generate light according to the image displayed in the fovea, it may be necessary to activate multiple rows simultaneously when operating multiple pixels (e.g., macropixels) in low-detail areas, and to activate individual rows when operating pixels in high-detail areas.

[0051] Figure 7 This is an example of modifying the phases of clock signals C1 to C8 according to some embodiments of this disclosure to simultaneously send gate signals to two rows. Figure 7 In the example, during the first display cycle H, clock signals C1 and C2 are both driven high, resulting in gate signal outputs at 421 and 422. Similarly, during the next display cycle H, C3 and C4 are both driven high, resulting in gate signal outputs at 423 and 424. It should be noted that due to the time overlap of the gate signals, [the output is different from the previous example]. Figure 6 Compared to the previous example, it can operate in half the time. Figure 6 The same number of display element rows. Therefore, as mentioned above, generating gate signals in this way can reduce the time required to generate image frames.

[0052] Alternatively, if a fixed image frame generation time is required when generating multiple gate signals simultaneously, then as follows: Figure 8As shown, a gate signal can be applied over multiple display cycles. Figure 8 In the example, the same as Figure 7 The same clock driving method, only the same signal is applied for a longer time, makes the total time for driving all rows of display elements using the gate signal longer than... Figure 6 The examples are the same.

[0053] Figure 9 This is an example of modifying the phases of clock signals C1 to C8 according to some embodiments of this disclosure to simultaneously send gate signals to four rows. Figure 9 In the example, during the first display cycle H, clock signals C1, C2, C3, and C4 are all driven high, which generates gate signal outputs at 421, 422, 423, and 424. Similarly, during the next display cycle H, clock signals C5, C6, C7, and C8 are all driven high, which generates gate signal outputs at 425, 426, 427, and 428.

[0054] Figure 10 This is an example of modifying the phases of clock signals C1 to C8 according to some embodiments of the present disclosure to send gate signals to different numbers of rows in consecutive display cycles. Figure 10 In the example, clock signals C1 and C2 are driven high in the first and second display cycles, respectively, generating gate signals at output 421 in the first display cycle and at output 422 in the second display cycle. In the next display cycle, clock signals C3 and C4 are both driven high, generating gate signal outputs at 423 and 424. In the next display cycle, clock signals C5, C6, C7, and C8 are all driven high, generating gate signal outputs at 425, 426, 427, and 428. Figure 10 The example illustrates how controlling the phase of each clock signal C1 through C8 can control the number of gate signals generated in the same display cycle, thereby controlling how many rows of display elements are driven simultaneously.

[0055] As described above, embodiments of this disclosure can provide a display system including a plurality of demultiplexers configured to simultaneously relay display data to one or more display element columns of the display, wherein the plurality of demultiplexers includes demultiplexers configured to simultaneously relay display data to a different number of columns. In contrast, other methods can be as follows: Figure 11 The operation is performed as shown, wherein during the display cycle H, display signals (DS) (DS1 to DS12) are provided sequentially to each column of display elements. Figure 11In the example, the display element is a subpixel, configured to generate red (R), green (G), or blue (B) light of a certain brightness when also driven by a gate signal, based on a received display signal. As described above, one method of displaying image frames is to drive each row (e.g., ...) sequentially in a continuous display cycle. Figure 3 As shown), and each column is driven sequentially during each display cycle, thereby addressing each display element one at a time. In some embodiments, different sub-pixels of each pixel (e.g., the red, green, and blue sub-pixels of the first pixel, which receive display signals DS1, DS2, and SD3 respectively) can be driven simultaneously. Therefore, Figure 11 The timeline depicted in the text can be compressed by a factor of three.

[0056] Figure 12 An improved method for driving sub-pixel columns is described when two columns need to be driven simultaneously. Figure 12 In the example, the six drive signals DSA, DSB, DSC, DSD, DSE, and DSF are each demultiplexed into two display signals to generate display signals DS1 to DS12. Each display signal DSA, DSB, DSC, DSD, DSE, and DSF is provided to a pair of sub-pixel columns. The physical arrangement of the sub-pixel columns may not necessarily be the same as... Figure 12 The matching shown Figure 12 This is provided only to illustrate how multiple display signals can be generated to simultaneously drive multiple sub-pixel columns. The driving signals, as well as the display signals generated by the demultiplexer, can be digital signals or analog signals representing digital values ​​(e.g., analog signals representing values ​​from 0 to 255 corresponding to the brightness of a sub-pixel).

[0057] In some embodiments, sub-pixel columns of different colors can be driven simultaneously. For example, a red sub-pixel column can be driven in the same time frame as a green and a blue sub-pixel column. Then, all other sub-pixel columns can be driven in subsequent time frames.

[0058] According to the technology described herein, a display system may include one or more demultiplexers, each demultiplexer being configured to demultiplex an input display signal into a plurality of identical output display signals and send these output display signals to a plurality of display element columns (e.g., subpixel columns). The following references... Figure 15An example of such a system is described. Each demultiplexer in the one or more demultiplexers can be configured to demultiplex the input display signal into a different number of output display signals. For example, a set of demultiplexers may include: a first demultiplexer configured to demultiplex the input display signal into two output display signals; a second demultiplexer configured to demultiplex the input display signal into three output display signals; a third demultiplexer configured to demultiplex the input display signal into four output display signals; and so on. The display system can also be configured to send the input display signal directly to the display element column when it is necessary to drive only a single column without driving other columns.

[0059] According to some embodiments, the display system can select one of the plurality of demultiplexers based on image data indicating one of the demultiplexers. The image data can be an analog signal or a digital signal. For example, the image data may include a digital value (e.g., corresponding to one of the plurality of demultiplexers), followed by image data describing the value used to drive that demultiplexer (e.g., a digital value from 0 to 255 indicating subpixel brightness), followed by another digital value corresponding to another demultiplexer (which may be the same or different demultiplexer), followed by image data describing another value used to drive that other demultiplexer, and so on. In some embodiments, the image data describing the value used to drive the selected demultiplexer may be a digital value converted to an analog signal and input to the demultiplexer, while the image data indicating which demultiplexer was selected is a digital value indicating which demultiplexer the display system selects. Alternatively, the image data may be an analog signal, wherein the portion of the analog signal indicating which demultiplexer was selected is a digital value, and the portion of the analog signal describes the value used to drive the selected demultiplexer.

[0060] Figure 13 Examples of a 1:2 demultiplexer according to some embodiments of this disclosure are depicted. Figure 13 In the example, input display signals can be provided to inputs R+, R-, G+, G-, B+, and / or B-, and the depicted circuit path, in conjunction with the operation of DMX1 switches, DMX2 switches, odd-frame switches, and even-frame switches, routes each input display signal to a pair of sub-pixel columns. As described below, Figure 13 The demultiplexer in the image is configured to drive each of the twelve sub-pixel columns depicted within two frames.

[0061] Figure 13The illustrative demultiplexer described is configured for a liquid crystal display (LCD) where at least alternating columns of subpixels have opposite polarities. Therefore, the signal provided to each subpixel is configured with a polarity suitable for the target subpixel.

[0062] operate Figure 13 An illustrative representation of the circuitry is as follows. In the first frame, the display signal R1 is provided positively to input 1301 and negatively to input 1302. In the first frame, the odd-numbered frame switch is open, and the even-numbered frame switch is closed. Additionally, the DMX1 switch is open, and the DMX2 switch is closed. Therefore, sub-pixel columns 1311 and 1314 receive the same brightness indicator R1, with sub-pixel column 1314 receiving the negative version and sub-pixel column 1311 receiving the positive version. Also in the first frame, the display signal G1 is provided positively to input 1303 and negatively to input 1304, such that sub-pixel columns 1312 and 1315 receive the same brightness indicator G1. Similarly, in the first frame, the display signal B1 is provided positively to input 1305 and negatively to input 1306, such that sub-pixel columns 1313 and 1316 receive the same brightness indicator B1. The result of this process is that, within a single frame, pairs of sub-pixel columns are driven by the same input values.

[0063] In the second frame, the odd-numbered frame switch is off, the even-numbered frame switch is on, the DMX1 switch is off, and the DMX2 switch is on. In this frame, display signals R2, G2, and B2 can be provided to inputs 1301 to 1306 with appropriate polarity to drive pairs of subpixel columns 1317 to 1322 with the same input values ​​in a single frame.

[0064] It is also conceivable that other modes, such as applying display signals to inputs 1301 to 1306, should be considered, since the above should not be regarded as being used for operation. Figure 13 The circuitry is constrained in this way. Similarly, the modes of switches DMX1 and DMX2 can be adjusted to change the routing behavior of the circuitry. In some cases, display signals with different values ​​(not just different polarities) can be provided to a pair of inputs associated with the same color (e.g., inputs 1301 and 1302, or inputs 1303 and 1304, or inputs 1305 and 1306) in the same frame. Therefore, in the same frame, multiple sub-pixel columns of the same color are not necessarily provided with the same brightness indication. However, in each configuration, the two sub-pixel columns of the same color will still be driven in the same frame.

[0065] Figure 14 Examples of a 1:3 demultiplexer according to some embodiments of this disclosure are depicted. Figure 14In the example, input display signals can be provided to inputs R+, R-, G+, G-, B+, and / or B-, and the depicted circuit path, in conjunction with the operation of DMX1, DMX2, DMX3, odd-frame, and even-frame switches, routes each input display signal to a pair of sub-pixel columns. As described below, Figure 14 The demultiplexer in the image is configured to drive each of the eighteen sub-pixel columns depicted within three frames.

[0066] operate Figure 14 An illustrative representation of the circuitry is as follows. In the first frame, display signal R1 is provided to inputs 1401 (positive) and 1402 (negative); display signal G1 is provided to inputs 1403 (negative) and 1404 (positive); display signal B1 is provided to inputs 1405 (positive) and 1406 (negative). In this frame, switch DMX1 is on, and switches DMX2 and DMX3 are off. In the second frame, display signal R1 is provided to input 1401 (positive), display signal R2 is provided to input 1402 (negative); display signal G1 is provided to input 1403 (negative), display signal G2 is provided to input 1404 (positive); display signal B1 is provided to input 1405 (positive), and display signal B2 is provided to input 1406 (negative). In this frame, DMX2 switch is on, and DMX1 and DMX3 switches are off. In the third frame, display signal R2 is provided to inputs 1401 (positive) and 1402 (negative); display signal G2 is provided to inputs 1403 (negative) and 1404 (positive); display signal B2 is provided to inputs 1405 (positive) and 1406 (negative). In this frame, DMX3 switch is off, and DMX1 and DMX2 switches are off.

[0067] and Figure 13 Similar to the circuit in the example, it is also conceivable that other modes of applying display signals to inputs 1401 to 1406 can be used, and the above should not be considered as being used for operation. Figure 14 The circuit's limiting method. Similarly, the modes of switches DMX1, DMX2, and DMX3 can be adjusted to change the circuit's routing behavior.

[0068] It can be envisioned that by following the description above and in Figure 13 and Figure 14 The techniques shown in the diagram are used for demultiplexing 1:4, 1:5, and other demultiplexers.

[0069] Figure 15An illustrative system 1500, operable according to some embodiments of the present disclosure and capable of operating according to the techniques described herein, is depicted. System 1500 may, for example, be part of a wearable device such as an artificial reality system, examples of which are described below.

[0070] exist Figure 15 In the example, system 1500 is partially controlled by system-on-a-chip (SoC) 1510, which generates image data for display on display 1540. Control of display 1540 is provided by display driver 1530, which receives image data from SoC 1510 and generates gate signals and display signals to address display elements in display 1540 as described above. Figure 15 In the example, the gate signal is generated by the gate circuit 1545, which can be implemented, for example, as follows: Figure 4 The gate circuit 400 shown is shown.

[0071] According to some embodiments, image data provided by SoC 1510 includes digital values ​​indicating the colors of multiple display elements of display 1540. The number of such digital values ​​for a single image frame may be less than the number of display elements in the display because, in a recessed display implemented as described herein, display signals can be provided simultaneously to multiple columns to address multiple display elements concurrently with multiple gate signals. Therefore, the size of the data stream representing the image frame can be smaller than that required to address each display element individually, potentially improving the efficiency of display rendering.

[0072] In some embodiments, the image data provided by SoC 1510 may further include information relating to the recessed layout of the image frame to be rendered, which informs controller 1532 how to operate the gate circuitry (e.g., how to control the frequency of the Nx clock of the gate circuitry) to simultaneously activate multiple rows of display 1540, at least in part, by simultaneously providing multiple gate signals to the display elements of the display, thereby rendering the image frame. In some embodiments, the information relating to the recessed layout of the image frame to be rendered may include the coordinate positions of various locations in the recessed layout within the image frame, such as the center of a region (e.g., a high-detail region, a medium-detail region, or a low-detail region), a corner of a region, etc. Additionally or alternatively, the information relating to the recessed layout of the image frame to be rendered may include the display element density of a particular region, the size of the region (e.g., horizontal and / or vertical dimensions), and / or any other information indicating which display elements should be addressed simultaneously and thus operated in the same manner.

[0073] exist Figure 15In the example, eye-tracking system 1520 can detect and measure the position of one or both eyes of the wearer of the device including system 1500. The following is combined with... Figure 25 and Figure 26 An illustrative eye-tracking system is described. Eye-tracking system 1520 can provide eye-tracking data, including indications of the position of one or both of a user's eyes, to SoC 1510. The SoC uses this eye-tracking data to generate image data for transmission to display driver 1530. For example, SoC 1510 can determine the location of high-detail regions of an image frame based on the position of one or both eyes and generate image data accordingly, as described above.

[0074] exist Figure 15 In the example, controller 1532 is configured to generate a clock signal to drive gate circuit 1545 (e.g., Figure 4 The gate circuit 1545 then outputs a gate signal to one or more display elements of the display 1540 via clock signal lines C1 to C8. The controller 1532 is also configured to control the display signal generator 1533 to generate display signals that, together with the gate signals generated by the gate circuit 1545, address the display elements in the display 1540. In some embodiments, the controller 1532 may (e.g., by operating the display signal generator according to the same clock signal period as the gate circuit 1545) generate the display signal and the gate signal synchronously. In some embodiments, the display signal generator 1533 may include a digital-to-analog converter configured to convert (e.g., received from the SoC 1510) digital values ​​indicating color or brightness into one or more analog display signals. For example, RGB digital values ​​may be converted into one or more (e.g., three) analog display signals that address the display elements (e.g., pixels or subpixels) of the display 1540. Although Figure 15 Not shown, but controller 1532 can also be configured to control which column of the columns the display signals are routed to.

[0075] Demultiplexer 1550 can receive display signals from display signal generator 1533 and demultiplex these display signals into multiple display signals as described above. Furthermore, as described above, display signal generator 1533 can (e.g., based on an indication of image data which demultiplexer to use to generate a portion of the display signal according to a given portion of the image data) route the display signals to a selected demultiplexer among the demultiplexers 1550. In some embodiments, in addition to one or more other demultiplexers, demultiplexer 1550 may also include Figure 13 and Figure 14 The demultiplexer shown.

[0076] although Figure 15 The various elements are described as separate subsystems, but it will be appreciated that these elements do not need to be implemented in this way. For example, the display driver 1530 may be implemented as an integrated circuit that at least implements the gate circuit 1545, the controller 1532, the display signal generator 1533, and the demultiplexer 1550. In some embodiments, the display driver 1530 may be regarded as a controller that performs the operations of the gate circuit 1545, the controller 1532, the display signal generator 1533, and the demultiplexer 1550 described above.

[0077] Based on the general principles described herein, features from any of the embodiments described herein can be combined with each other. These and other embodiments, features, and advantages will be more fully understood upon reading the following detailed description in conjunction with the accompanying drawings and claims.

[0078] It will be understood that, as used herein, "same time" or "simultaneously" driving multiple rows or columns of display elements means that each row and / or each column is addressed simultaneously. The signals driving the rows and / or columns do not necessarily need to start or end simultaneously to be considered "same time" or "simultaneously" generated. For example, two signals that both trigger on their rising edges can end at different times without affecting their simultaneous addressing. Similarly, two signals that both trigger on their trailing edges can start at different times without affecting their simultaneous addressing.

[0079] Example Implementation

[0080] Example 1. A system comprising: a plurality of addressable display elements arranged in a plurality of display element rows and a plurality of display element columns; a plurality of clock signal lines; a plurality of gate driving units, each gate driving unit coupled to one or more of the plurality of clock signal lines and configured to send a gate signal to one or more display element rows of the plurality of display element rows based on a clock signal provided on one or more clock signal lines; and a controller configured to generate a clock signal such that the plurality of gate driving units send gate signals to corresponding display element rows of the plurality of display element rows, wherein the clock signal is controlled to adjust the number of gate signals simultaneously sent to the corresponding display element rows according to a recessed display pattern.

[0081] Example 2. According to the system of Example 1, wherein the controller is configured to: generate a clock signal during a first display cycle, thereby causing a first gate driving unit among the plurality of gate driving units to send a gate signal to one or more display element rows among the plurality of display element rows during the first display cycle; and send a corresponding display signal to each display element column among the plurality of display element columns during the first display cycle.

[0082] Example 3. A system according to any one of Examples 1 to 2, wherein the controller is configured to, when the central recessed display pattern includes macro pixels: generate a clock signal during a first display cycle, such that a first gate driving unit of the plurality of gate driving units simultaneously sends gate signals to at least two display element rows of the plurality of display element rows during the first display cycle; and simultaneously sends corresponding display signals to at least two display element columns of the plurality of display element columns during the first display cycle.

[0083] Example 4. A system according to any of Examples 1 to 3, wherein the corresponding display signals sent simultaneously to at least two of the plurality of display element columns during a first display cycle are the same display signals.

[0084] Example 5. A system according to any of Examples 1 to 4, wherein: a first gate driving unit of the plurality of gate driving units simultaneously sends gate signals to four display element rows of the plurality of display element rows during a first display cycle; and the controller is further configured to simultaneously send the same display signal to four display element columns of the plurality of display element columns during the first display cycle.

[0085] Example 6. A system according to any one of Examples 1 to 5, wherein the plurality of gate driving units include: a first gate driver having a plurality of clock inputs coupled to a first subset of a plurality of clock signal lines; and a second gate driver having a plurality of clock inputs coupled to a second subset of a plurality of clock signal lines, the second subset being different from the first subset.

[0086] Example 7. A system according to any of Examples 1 to 6, wherein at least one of the plurality of clock inputs of the respective gate driver in the plurality of gate drive units enables or disables the function of the respective gate driver with respect to transmitting one or more gate signals.

[0087] Example 8. A system according to any of Examples 1 to 7, wherein the plurality of addressable display elements comprises a plurality of pixels and / or a plurality of subpixels.

[0088] Example 9. A system according to any of Examples 1 to 8, wherein the controller, the plurality of gate drive units, and the plurality of clock signal lines are implemented as integrated circuits.

[0089] Example 10. A method comprising: simultaneously transmitting gate signals by a first gate driving unit to each of a first number of rows of a plurality of display elements, the first gate driving unit receiving first clock signals from a first subset of a plurality of clock signal lines; and simultaneously transmitting gate signals by a second gate driving unit to each of a second number of rows of the plurality of display elements, the second gate driving unit receiving second clock signals from a second subset of the plurality of clock signal lines, the second subset being different from the first subset, wherein the second number of rows is different from the first number of rows.

[0090] Example 11. The method according to Example 10 further includes: adjusting the phase of the first clock signal to generate a second clock signal.

[0091] Example 12. The method according to any of Examples 10 to 11 further includes: the controller generating a first clock signal and a second clock signal based on the image data.

[0092] Example 13. A method according to any of Examples 10 to 12, wherein the plurality of display elements are arranged into a plurality of display element columns and a plurality of display element rows; and wherein the method further comprises: directing a gate signal to each of a first number of rows of the plurality of display elements, while directing a display signal to one or more display element columns of the plurality of display element columns, thereby simultaneously directing both the display signal and the gate signal to one or more display elements.

[0093] Example 14. A system comprising: a plurality of addressable display elements arranged in a plurality of display element rows and a plurality of display element columns; and a controller including a plurality of demultiplexers and configured to perform the following operations based on image data received by the controller: a first demultiplexer of the plurality of demultiplexers simultaneously relays first display data to a first number of display element columns in the plurality of display element columns; and after relaying the first display data to the first number of display element columns in the plurality of display element columns, a second demultiplexer of the plurality of demultiplexers simultaneously relays second display data to a second number of display element columns in the plurality of display element columns, the second number of display element columns being different from the first number of display element columns in the plurality of display element columns.

[0094] Example 15. The system according to Example 14, wherein the image data includes a plurality of demultiplexing indicators; and wherein the controller is configured to select one of the plurality of demultiplexers to receive the display data based on one of the demultiplexing indicators.

[0095] Example 16. A system according to any of Examples 14 to 15, wherein the controller is configured to send first display data to a first demultiplexer based on image data including a first demultiplexer indicator, first image data, a second demultiplexer indicator, and second image data, and to send second display data to a second demultiplexer based on the second demultiplexer indicator and the second image data.

[0096] Example 17. A system according to any of Examples 14 to 16, wherein the first number of the plurality of display element columns is greater than 1; and wherein the second number of the plurality of display element columns is 1.

[0097] Example 18. A system according to any of Examples 14 to 17, wherein the controller is configured to generate first display data and second display data based on image data, at least in part, using a digital-to-analog converter.

[0098] Example 19. A system according to any of Examples 14 to 18, wherein the plurality of display element columns are configured with alternating polarities; and wherein the first demultiplexer includes: a first input configured to route a first portion of first display data to a first display element column of the plurality of display element columns; and a second input configured to route a second portion of the first display data to a second display element column of the plurality of display element columns, the second display element column being adjacent to the first display element column and having a polarity opposite to that of the first display element column.

[0099] Example 20. A system based on any of Examples 14 to 19, wherein the controller is configured to generate a first portion of first display data, the first portion of the first display data being a polarity-reversed version of a second portion of the first display data.

[0100] Example 21. A system according to any of Examples 14 to 20, wherein the first display data and the second display data are analog data signals.

[0101] Example 22. A method comprising: simultaneously relaying first display data to a first number of display element columns of a plurality of display element columns by a first demultiplexer of a plurality of demultiplexers; and after relaying the first display data to the first number of display element columns of the plurality of display element columns, simultaneously relaying second display data to a second number of display element columns of the plurality of display element columns by a second demultiplexer of the plurality of demultiplexers, the second number of display element columns of the plurality of display element columns being different from the first number of display element columns of the plurality of display element columns.

[0102] Example 23. According to the method of Example 22, wherein the first demultiplexer and the second demultiplexer are implemented by different circuit sections within the display driver integrated circuit.

[0103] Example 24. The method according to any of Examples 22 to 23 further includes: the controller selecting one of a plurality of demultiplexers, including a first demultiplexer and a second demultiplexer, to receive display data based on image data including a plurality of demultiplexing indicators and one of the plurality of demultiplexing indicators.

[0104] Example 25. The method according to any of Examples 22 to 24, wherein the first number of the plurality of display element columns is greater than 1; and wherein the second number of the plurality of display element columns is 1.

[0105] Example 26. A method according to any of Examples 22 to 25, wherein relaying first display data to a first number of display element columns among the plurality of display element columns comprises: relaying the first display data to a plurality of display element columns configured to produce light of the same color.

[0106] Example 27. The method of any of Examples 22 to 26, wherein the color of the light is red, green or blue.

[0107] The embodiments of this disclosure may include various types of Artificial-Reality (AR) systems, or may be implemented by combining various types of AR systems. AR can be any overlay functionality and / or sensorially detectable content presented by an AR system within a user's physical environment. In other words, AR is a form of reality that has been adjusted in some way before being presented to the user. AR may include and / or represent virtual reality (VR), augmented reality, mixed AR (MAR), or some combination and / or variation of these types of reality. Similarly, AR environments may include VR environments (including non-immersive VR environments, semi-immersive VR environments, and fully immersive VR environments), augmented reality environments (including marker-based augmented reality environments, markerless augmented reality environments, location-based augmented reality environments, and projection-based augmented reality environments), mixed reality environments, and / or any other type or form of mixed reality environment or alternative reality environment.

[0108] AR content can consist entirely of computer-generated content or include computer-generated content combined with acquired (e.g., real-world) content. AR content can include video, audio, haptic feedback, or some combination thereof, any of which can be presented in a single channel or multiple channels (e.g., stereoscopic video providing a three-dimensional (3D) effect to the viewer). Additionally, in some embodiments, AR can be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or otherwise for use in artificial reality (e.g., to perform activities in artificial reality).

[0109] AR systems can be implemented in a variety of different shapes and configurations. Some AR systems can be designed to operate without a near-eye display (NED). Other AR systems may include NEDs, which also provide visibility into the real world (e.g., Figure 22 Augmented reality systems (also known as head-mounted wearable devices or AR glasses) 2200) or those that allow users to visually immerse themselves in artificial reality (e.g., Figure 23A and Figure 23BVirtual reality systems (also known as head-mounted wearable devices or VR headsets) in the context of AR (2310). While some AR devices may be standalone systems, others may communicate with and / or cooperate with external devices to provide an AR experience to the user. Examples of such external devices include handheld controllers, mobile devices, desktop computers, devices worn by the user, devices worn by one or more other users, and / or any other suitable external system.

[0110] Figures 16 to 19B An example artificial reality (AR) system according to some embodiments is shown. Figure 16 The first AR system 1600 and a first example user interaction are shown, which uses a wrist wearable device 1602, a head wearable device (e.g., AR glasses 1604) and / or a handheld intermediate processing device (HIPD) 1606. Figure 17 A second AR system 1700 and a second example user interaction are shown, which uses a wrist wearable device 1702, AR glasses 1704 and / or HIPD 1706. Figure 18A and Figure 18B The interaction between a third AR system 1800 and a third example user 1808 is shown, using a wrist wearable device 1802, a head wearable device (e.g., a VR headset 1850), and / or a HIPD 1806. Figure 19A and Figure 19B The interaction between the fourth AR system 1900 and the fourth example user 1908 is shown, which uses a wrist wearable device 1930, a VR headset 1920 and / or a haptic device 1960 (e.g., wearable gloves).

[0111] The following is for reference Figure 20 and Figure 21 This describes a wrist-worn wearable device 2000 and one or more of its components. The wrist-worn wearable device 2000 can be used in wrist-worn wearable devices 1602, 1702, 1802, and 1930; the following references... Figures 22 to 24 The description includes head-worn devices 2200 and 2310 and one or more components thereof, which can be used in AR glasses 1604 and 1704 or VR headsets 1850 and 1920, respectively.

[0112] refer to Figure 16The wrist-worn wearable device 1602, AR glasses 1604, and / or HIPD 1606 can be communicatively coupled via a network 1625 (e.g., cellular, near-field, Wi-Fi, personal area network, wireless local area network (LAN), etc.). Additionally, the wrist-worn wearable device 1602, AR glasses 1604, and / or HIPD 1606 can also be communicatively coupled via the network 1625 (e.g., cellular, near-field, Wi-Fi, personal area network, wireless LAN, etc.) to one or more servers 1630, computers 1640 (e.g., laptops, computers, etc.), mobile devices 1650 (e.g., smartphones, tablets, etc.), and / or other electronic devices.

[0113] exist Figure 16 The image shows a user 1608 wearing a wrist-worn wearable device 1602 and AR glasses 1604, with a HIPD 1606 placed on their table. The wrist-worn wearable device 1602, AR glasses 1604, and HIPD 1606 facilitate the user's interaction with the AR environment. Specifically, as shown in the first AR system 1600, the wrist-worn wearable device 1602, AR glasses 1604, and / or HIPD 1606 enable the presentation of one or more avatars 1610, digital representations of contacts 1612, and virtual objects 1614. As discussed below, the user 1608 can interact with one or more avatars 1610, digital representations of contacts 1612, and virtual objects 1614 via the wrist-worn wearable device 1602, AR glasses 1604, and / or HIPD 1606.

[0114] User 1608 may use any of the wrist-worn wearable device 1602, AR glasses 1604, and / or HIPD 1606 to provide user input. For example, user 1608 may perform one or more gestures to provide user input, which may be performed by the wrist-worn wearable device 1602 (e.g., using the following reference). Figure 20 and Figure 21 The description describes one or more electromyography (EMG) sensors and / or IMUs that detect and / or are used by AR glasses 1604 (e.g., using the following references). Figures 22 to 24The detection is performed by one or more image sensors or cameras. Alternatively or additionally, user 1608 may provide user input via one or more touch surfaces of the wrist wearable device 1602, AR glasses 1604, and / or HIPD 1606, and / or voice commands acquired by the microphones of the wrist wearable device 1602, AR glasses 1604, and / or HIPD 1606. In some embodiments, the wrist wearable device 1602, AR glasses 1604, and / or HIPD 1606 includes a digital assistant for assisting user 1608 in providing user input (e.g., performing a series of actions, recommending different actions or commands, providing reminders, confirming commands, etc.). In some embodiments, user 1608 may provide user input via one or more facial gestures and / or facial expressions. For example, the cameras of the wrist wearable device 1602, AR glasses 1604, and / or HIPD 1606 may track user 1608's eyes for navigating the user interface.

[0115] The wrist-worn wearable device 1602, AR glasses 1604, and / or HIPD 1606 can operate individually or in combination to allow user 1608 to interact with the AR environment. In some embodiments, HIPD 1606 is configured to operate as a central hub or control center for the wrist-worn wearable device 1602, AR glasses 1604, and / or another communication-coupled device. For example, user 1608 can provide input for interacting with the AR environment at any of the wrist-worn wearable device 1602, AR glasses 1604, and / or HIPD 1606, and HIPD 1606 can identify one or more backend and frontend tasks to perform the requested interaction and distribute instructions to execute the one or more backend and frontend tasks at the wrist-worn wearable device 1602, AR glasses 1604, and / or HIPD 1606. In some embodiments, backend tasks are user-invisible background processing tasks (e.g., rendering content, decompressing, compressing, etc.), while frontend tasks are user-perceptible user-facing tasks (e.g., presenting information to the user, providing feedback to the user, etc.). The HIPD 1606 can perform backend tasks and provide operation data corresponding to the performed backend tasks to the wrist wearable device 1602 and / or AR glasses 1604, enabling the wrist wearable device 1602 and / or AR glasses 1604 to perform frontend tasks. In this way, the HIPD 1606 (which has more computing resources and greater thermal headroom than the wrist wearable device 1602 and / or AR glasses 1604) performs computationally intensive tasks and reduces the computing resource utilization and / or power consumption of the wrist wearable device 1602 and / or AR glasses 1604.

[0116] In the example shown in the first AR system 1600, HIPD 1606 identifies one or more backend and frontend tasks associated with a user request initiating an AR video call with one or more other users (represented by avatar 1610 and digital representations of contacts 1612), and issues instructions to execute the one or more backend and frontend tasks. Specifically, HIPD 1606 performs backend tasks for processing and / or rendering image data (and other data) associated with the AR video call, and provides AR glasses 1604 with operational data associated with the performed backend tasks, causing AR glasses 1604 to perform frontend tasks for presenting the AR video call (e.g., presenting avatar 1610 and digital representations of contacts 1612).

[0117] In some embodiments, HIPD 1606 can function as a focal point or anchor point for information presentation. This allows user 1608 to know where the information is presented. For example, as shown in the first AR system 1600, avatar 1610 and digital representations 1612 of contacts are presented above HIPD 1606. Specifically, HIPD 1606 and AR glasses 1604 operate in combination to determine the location for presenting avatar 1610 and digital representations 1612 of contacts. In some embodiments, information can be presented within a predetermined distance from HIPD 1606 (e.g., within 5 meters). For example, as shown in the first AR system 1600, virtual object 1614 is presented on a table at a distance from HIPD 1606. Similar to the above examples, HIPD 1606 and AR glasses 1604 can operate in combination to determine the location for presenting virtual object 1614. Alternatively, in some embodiments, the presentation of information is not constrained by HIPD 1606. More specifically, the avatar 1610, the digital representation of the contact 1612, and the virtual object 1614 need not be presented within the predetermined distance of the HIPD 1606.

[0118] The system coordinates user input provided at the wrist-worn wearable device 1602, AR glasses 1604, and / or HIPD 1606, enabling a user to initiate, continue, and / or complete an operation using any device. For example, user 1608 may provide user input to AR glasses 1604 to cause AR glasses 1604 to display a virtual object 1614, and while AR glasses 1604 displays the virtual object 1614, user 1608 may provide one or more gestures via the wrist-worn wearable device 1602 to interact with and / or manipulate the virtual object 1614.

[0119] Figure 17The image shows user 1708 wearing wrist-worn wearable device 1702 and AR glasses 1704, and holding HIPD 1706. In the second AR system 1700, wrist-worn wearable device 1702, AR glasses 1704, and / or HIPD 1706 are used to receive one or more messages and / or provide one or more messages to user 1708's contacts. Specifically, wrist-worn wearable device 1702, AR glasses 1704, and / or HIPD 1706 detect and coordinate one or more user inputs to launch a messaging application, and prepare a response to the received message via that messaging application.

[0120] In some embodiments, user 1708 initiates an application on wrist wearable device 1702, AR glasses 1704, and / or HIPD 1706 via user input, thereby launching the application on at least one device. For example, in a second AR system 1700, user 1708 performs a gesture associated with a command to launch a messaging application (represented by messaging user interface 1716); wrist wearable device 1702 detects the gesture and, based on determining that user 1708 is wearing AR glasses 1704, causes AR glasses 1704 to present the messaging user interface 1716 of the messaging application. AR glasses 1704 may present the messaging user interface 1716 to user 1708 via its display (e.g., as shown in user 1708's field of view 1718). In some embodiments, the application is launched and executed on a device (e.g., wrist wearable device 1702, AR glasses 1704, and / or HIPD 1706) that detects user input for launching the application, and that device provides operational data to another device to present the messaging application. For example, the wrist-worn wearable device 1702 can detect user input for launching a messaging application, launch and run the messaging application, and provide operational data to the AR glasses 1704 and / or HIPD 1706 to render the messaging application. Alternatively, the application can be launched and executed on a different device than the one that detected the user input. For example, the wrist-worn wearable device 1702 can detect gestures associated with launching the messaging application and can enable the HIPD 1706 to run the messaging application and coordinate the rendering of the messaging application.

[0121] Furthermore, user 1708 can provide user input at the wrist wearable device 1702, AR glasses 1704, and / or HIPD 1706 to continue and / or complete an operation initiated at another device. For example, after launching a messaging application via the wrist wearable device 1702 and when the messaging user interface 1716 is presented on the AR glasses 1704, user 1708 can provide input at the HIPD 1706 to prepare a response (e.g., indicated by a swipe gesture performed on the HIPD 1706). The gesture performed by user 1708 on the HIPD 1706 can be provided and / or displayed on another device. For example, a swipe gesture performed on the HIPD 1706 is displayed on the virtual keyboard of the messaging user interface 1716 displayed by the AR glasses 1704.

[0122] In some embodiments, the wrist wearable device 1702, AR glasses 1704, HIPD 1706, and / or other communication-coupled devices may present one or more notifications to the user 1708. The notification may be an indication of a new message, incoming call, application update, status update, etc. The user 1708 may select a notification via the wrist wearable device 1702, AR glasses 1704, or HIPD 1706, causing the application or action associated with the notification to be presented on at least one device. For example, user 1708 may receive a notification of receiving a message at wrist wearable device 1702, AR glasses 1704, HIPD 1706 and / or other communication-coupled devices, and provide user input at wrist wearable device 1702, AR glasses 1704 and / or HIPD 1706 to view the notification, and the device that detects the user input may enable the launch of an application associated with the notification and / or present an application associated with the notification at wrist wearable device 1702, AR glasses 1704 and / or HIPD 1706.

[0123] While the examples above describe coordinated input for interacting with messaging applications, user input can be coordinated to interact with any number of applications, including but not limited to gaming applications, social media applications, camera applications, web-based applications, financial applications, etc. For example, AR glasses 1704 can present game application data to user 1708, while HIPD 1706 can use a controller to provide input to the game. Similarly, user 1708 can use wrist wearable device 1702 to activate the camera of AR glasses 1704, and user 1708 can use wrist wearable device 1702, AR glasses 1704, and / or HIPD 1706 to manipulate image acquisition (e.g., zoom in or out, apply filters, etc.) and acquire image data.

[0124] Users can interact with the device described in this article in a variety of ways. For example, such as Figure 18A and Figure 18B As shown, user 1808 can interact with AR system 1800 by wearing VR headset 1850, holding HIPD 1806, and wearing wrist wearable device 1802. In this example, AR system 1800 allows the user to interact with game 1010 by waving their arm. One or more of VR headset 1850, HIPD 1806, and wrist wearable device 1802 can detect the gesture and, in response, display a sword strike in game 1010. Similarly, in Figure 19A and Figure 19B In this example, user 1908 can interact with AR system 1900 by wearing VR headset 1920, haptic device 1960, and wrist wearable device 1930 simultaneously. In this example, AR system 1900 allows the user to interact with game 1110 by waving their arm. One or more of the VR headset 1920, haptic device 1960, and wrist wearable device 1930 can detect the gesture and, in response, display the casting of a spell in game 1810.

[0125] Having discussed the example AR system, this paper will now discuss in more detail the devices used to interact with such AR systems and other computing systems in a wider range. For ease of reference, this paper provides some explanations of the devices and components that may be included in some or all of the example devices discussed below. Certain types of parts described below may be more suitable for a particular set of devices and less suitable for different sets of devices. However, subsequent references to parts defined herein should be considered as included in the description provided.

[0126] In some of the embodiments discussed below, several example devices and systems, including electronic devices and systems, will be discussed. These example devices and systems are not intended to be limiting, and those skilled in the art will understand that alternative devices and systems to the example devices and systems described herein can be used to perform the operations described herein and to construct the systems and devices described herein.

[0127] An electronic device can be a device that uses electrical energy to perform a specific function. An electronic device can be any physical object that contains electronic components such as transistors, resistors, capacitors, diodes, and integrated circuits. Examples of electronic devices include smartphones, laptops, digital cameras, televisions, game consoles, and music players, as well as the various example electronic devices discussed herein. As described herein, an intermediate electronic device can be a device located between two other electronic devices and / or between subsets of multiple components of one or more electronic devices, facilitating communication, data processing, and / or data transmission between the respective electronic devices and / or electronic components.

[0128] An integrated circuit (IC) can be an electronic device composed of multiple interconnected electronic components, such as transistors, resistors, and capacitors. These components can be etched onto a small piece of semiconductor material, such as silicon. ICs can include analog ICs, digital ICs, mixed-signal ICs, and / or any other suitable type or form of IC. Examples of ICs include application-specific integrated circuits (ASICs), processing units, central processing units (CPUs), coprocessors, and accelerators.

[0129] Analog integrated circuits (such as sensors, power management circuits, and operational amplifiers) can process continuous signals and perform analog functions such as amplification, active filtering, demodulation, and mixing. Examples of analog integrated circuits include linear integrated circuits and radio frequency circuits.

[0130] Digital integrated circuits (which may be referred to as logic integrated circuits) may include microprocessors, microcontrollers, memory chips, interfaces, power management circuits, programmable devices, and / or any other suitable type or form of integrated circuit. In some embodiments, examples of integrated circuits include central processing units (CPUs).

[0131] Processing units, such as CPUs, can be electronic components responsible for executing instructions and controlling the operation of electronic devices (e.g., computers). Various types of processors exist, which can be used interchangeably or may be specifically required by the embodiments described herein. For example, a processor can be: (i) a general-purpose processor designed to perform a wide range of tasks, such as running software applications, managing operating systems, and performing arithmetic and logical operations; (ii) a microcontroller designed for specific tasks, such as controlling electronic devices, sensors, and motors; (iii) an accelerator, such as a graphics processing unit (GPU) designed to accelerate the creation and rendering of images, videos, and animations (e.g., virtual reality animation, such as 3D modeling); (iv) a field-programmable gate array (FPGA) that can be programmed and reconfigured post-manufacturing and / or customized to perform specific tasks, such as signal processing, encryption, and machine learning; and / or (v) a digital signal processor (DSP) designed to perform mathematical operations on signals (e.g., audio, video, and radio waves). One or more processors of one or more electronic devices may be used in the various embodiments described herein.

[0132] Memory broadly refers to electronic components in a computer or electronic device that store data and instructions for access and manipulation by a processor. Examples of memory may include: (i) random access memory (RAM) configured to temporarily store data and instructions; (ii) read-only memory (ROM) configured to permanently store data and instructions (e.g., one or more portions of system firmware and / or a bootloader) and / or semi-permanently store data and instructions; (iii) flash memory configured to store data in an electronic device (e.g., a universal serial bus (USB) drive, a memory card, and / or a solid-state drive (SSD)); and / or (iv) cache memory configured to temporarily store frequently accessed data and instructions. As described herein, memory may store structured data (e.g., SQL databases, MongoDB databases, GraphQL data, JSON data, etc.). Other examples of data stored in memory may include: (i) data data, including user account data, user settings, and / or other user data stored by the user; (ii) sensor data detected by one or more sensors and / or otherwise acquired; (iii) media content data, including stored image data, audio data, and documents; (iv) application data, which may include data collected and / or otherwise acquired and stored during the use of the application; and / or any other types of data described herein.

[0133] A controller can be an electronic component that manages and coordinates the operation of other components within an electronic device (e.g., controlling inputs, processing data, and / or generating outputs). Examples of controllers may include: (i) microcontrollers, which include small, low-power controllers commonly used in embedded systems and Internet of Things (IoT) devices; (ii) programmable logic controllers (PLCs), which can be configured for use in industrial automation systems to control and monitor manufacturing processes; (iii) system-on-a-chip (SoC) controllers, which integrate multiple components such as processors, memory, input / output (I / O) interfaces, and other peripherals into a single chip; and / or (iv) digital signal processors (DSPs).

[0134] The power system of an electronic device can be configured to convert input power into a form usable for operating the device. The power system may include various components, such as: (i) a power source, which may be an alternating current (AC) adapter power source or a direct current (DC) adapter power source; (ii) a charger input, which may be configured to use wired and / or wireless connections (this charger input may be part of a peripheral interface, such as USB, microUSB, near-field magnetic coupling, magnetic induction and magnetic resonance charging, and / or radio frequency (RF) charging); (iii) a power management integrated circuit configured to distribute power to the various components of the device and ensure that the device operates within safe limits (e.g., regulating voltage, controlling current, and / or managing heat dissipation); and / or (iv) a battery configured to store power to provide usable power to the components of one or more electronic devices.

[0135] Peripheral interfaces can be electronic components (e.g., electronic components of an electronic device) that allow the electronic device to communicate with other devices or peripheral devices and can provide the ability to input and output data and signals. Examples of peripheral interfaces may include: (i) a Universal Serial Bus (USB) interface and / or a microUSB interface configured to connect a device to an electronic device; (ii) a Bluetooth interface configured to allow multiple devices to communicate with each other, including Bluetooth Low Energy (BLE); (iii) a near-field communication (NFC) interface configured as a short-range wireless interface for operations such as access control; (iv) a POGO pin, which may be a small, spring-loaded pin configured to provide a charging interface; (v) a wireless charging interface; (vi) a global positioning system (GPS) interface; (vii) a Wi-Fi interface for providing connectivity between the device and a wireless network; and / or (viii) a sensor interface.

[0136] Sensors can be electronic components configured to detect physical and environmental changes and generate electrical signals (e.g., electronic components in electronic devices (e.g., wearable devices) and / or electronic components that otherwise communicate electronically with electronic devices). Examples of sensors may include: (i) imaging sensors for collecting imaging data (e.g., including one or more cameras disposed on a corresponding electronic device); (ii) biopotential signal sensors; (iii) inertial measurement units (e.g., IMUs) for detecting, for example, changes in angular velocity, force, magnetic field, and / or acceleration; (iv) heart rate sensors for measuring a user's heart rate; (v) blood oxygen saturation (SpO2) sensors for measuring a user's blood oxygen saturation and / or other biometric data; (vi) capacitive sensors for detecting potential changes at a site on a user's body (e.g., a sensor-skin interface); and / or (vii) light sensors (e.g., time-of-flight sensors, infrared sensors, visible light sensors, etc.).

[0137] Biopotential signal sensing components can be devices used to measure electrical activity within the body (e.g., biopotential signal sensors). Some types of biopotential signal sensors include: (i) electroencephalography (EEG) sensors configured to measure electrical activity in the brain to diagnose neurological disorders; (ii) electrocardiography (ECG or EKG) sensors configured to measure electrical activity in the heart to diagnose heart problems; (iii) electromyography (EMG) sensors configured to measure electrical activity in muscles and diagnose neuromuscular disorders; and (iv) electrooculography (EOG) sensors configured to measure electrical activity in eye muscles to detect eye movements and diagnose eye disorders.

[0138] Applications (e.g., software) stored in the memory of an electronic device may include instructions stored in the memory. Examples of such applications include: (i) games; (ii) word processors; (iii) messaging applications; (iv) media streaming applications; (v) financial applications; (vi) calendars; (vii) clocks; and (viii) communication interface modules for enabling wired and / or wireless connections between different corresponding electronic devices (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, or MiWi; custom wired protocols or standard wired protocols (e.g., Ethernet or HomePlug); and / or any other suitable communication protocol).

[0139] A communication interface can be a mechanism that enables different systems or devices to exchange information and data with each other, including hardware, software, or a combination of both. For example, a communication interface can refer to a physical connector and / or port on a device that enables communication with other devices (e.g., USB, Ethernet, High Definition Multimedia Interface (HDMI), Bluetooth). In some embodiments, a communication interface can refer to a software layer that enables different software programs to communicate with each other (e.g., application programming interface (API), protocols such as Hypertext Transfer Protocol (HTTP) and Transmission Control Protocol / Internet Protocol (TCP / IP)).

[0140] A graphics module can be a component or software module designed to handle graphics operations and / or graphical processes, and the graphics module may include hardware modules and / or software modules.

[0141] Non-transitory computer-readable storage media can be physical devices or storage media that can be used to store electronic data in a non-transitory form (e.g., such that the data is permanently stored until it is intentionally deleted or modified).

[0142] Figure 20 and Figure 21 An example wrist-worn wearable device 2000 and an example computing system 2100 are illustrated according to some embodiments. The wrist-worn wearable device 2000 is described herein. Figure 16 The wearable device 1602 described herein is such that the wearable device 1602 should be understood as having the characteristics of the wrist wearable device 2000, and vice versa. Figure 21Several components of a wrist-worn wearable device 2000 are shown, which can be used individually or in combination, including combinations that include other electronic devices and / or electronic components.

[0143] Figure 20 The image shows a wearable strap 2010 and a watch body 2020 (or capsule) coupled together to form a wrist-worn wearable device 2000. The wrist-worn wearable device 2000 can perform various functions and / or operations associated with navigating in a user interface and selectively activating applications, as well as the above references. Figures 16 to 19B The described functions and / or operations.

[0144] As will be described in more detail below, the operations performed by the wrist-worn wearable device 2000 may include: (i) presenting content to a user (e.g., displaying visual content via a display 2005); (ii) detecting (e.g., sensing) user input (e.g., sensing touches on peripheral buttons 2023 and / or touches on the touchscreen of the display 2005, sensing gestures detected by sensors (e.g., biopotential sensors); (iii) sensing biometric data (e.g., neuromuscular signals, heart rate, temperature, sleep, etc.) via one or more sensors 2013; messaging (e.g., text, voice, video, etc.); image acquisition via one or more imaging devices or cameras 2025; wireless communication (e.g., cellular, near-field, Wi-Fi, personal area network, etc.); location determination; financial transactions; providing haptic feedback; providing alarms; providing notifications; providing biometric authentication; providing health monitoring; providing sleep monitoring, etc.

[0145] The example functions described above can be performed independently in the watch body 2020, independently in the wearable band 2010, and / or via electronic communication between the watch body 2020 and the wearable band 2010. In some embodiments, the functions can be performed on the wrist wearable device 2000 when an AR environment is presented (e.g., via one of the AR systems 1600 to 1900). The wearable device described herein can also be used with other types of AR environments.

[0146] The wearable band 2010 can be configured to be worn by a user such that the inner surface of the wearable structure 2011 of the wearable band 2010 contacts the user's skin. In this example, the sensor 2013 can contact the user's skin when worn by the user. In some examples, one or more of the sensors 2013 can sense biometric data, such as the user's heart rate, saturated oxygen level, temperature, sweat level, neuromuscular signals, or combinations thereof. One or more of the sensors 2013 can also sense data about the user's environment, including the user's motion, altitude, location, orientation, gait, acceleration, position, or combinations thereof. In some embodiments, one or more of the sensors 2013 can be configured to track the position and / or motion of the wearable band 2010. One or more of the sensors 2013 can include the features defined above and / or the following regarding... Figure 20 Any of the multiple sensors discussed.

[0147] One or more sensors of each sensor 2013 may be distributed on the inner and / or outer surface of the wearable band 2010. In some embodiments, one or more sensors of each sensor 2013 are evenly spaced along the wearable band 2010. Alternatively, in some embodiments, one or more sensors of each sensor 2013 are located at different points along the wearable band 2010. Figure 20 As shown, one or more sensors in each of the sensors 2013 may be the same or different. For example, in some embodiments, one or more sensors in each of the sensors 2013 may be shaped as a pill (e.g., sensor 2013a), oval, circular, square, elliptical (e.g., sensor 2013c), and / or any other shape that maintains contact with the user's skin (e.g., so that neuromuscular signals and / or other biometric data can be accurately measured at the user's skin). In some embodiments, one or more sensors in each of the sensors 2013 are aligned to form sensor pairs (e.g., for sensing neuromuscular signals based on differential sensing within each respective sensor). For example, sensor 2013b may be aligned with an adjacent sensor to form sensor pair 2014a, and sensor 2013d may be aligned with an adjacent sensor to form sensor pair 2014b. In some embodiments, the wearable band 2010 does not have sensor pairs. Alternatively, in some embodiments, the wearable band 2010 has a predetermined number of sensor pairs (one sensor pair, three sensor pairs, four sensor pairs, six sensor pairs, sixteen sensor pairs, etc.).

[0148] The wearable band 2010 may include any suitable number of sensors 2013. In some embodiments, the number and arrangement of the sensors 2013 depend on the specific application using the wearable band 2010. For example, the wearable band 2010 may be configured as an armband, wristband, or chest band including multiple sensors 2013, with each use case (e.g., a medical use case) having a different number of sensors 2013, individual sensors of various types among the multiple sensors 2013, and different arrangements compared to gaming use cases or general everyday use cases.

[0149] According to some embodiments, the wearable band 2010 also includes an electrically grounding electrode and a shielding electrode. Similar to the sensor 2013, the electrically grounding electrode and the shielding electrode may be distributed on the inner surface of the wearable band 2010 such that they contact a portion of the user's skin. For example, the electrically grounding electrode and the shielding electrode may be located on the inner surface of the coupling mechanism 2016 or the inner surface of the wearable structure 2011. The electrically grounding electrode and the shielding electrode may be formed and / or use the same components as those in the sensor 2013. In some embodiments, the wearable band 2010 includes more than one electrically grounding electrode and more than one shielding electrode.

[0150] Sensors 2013 may be formed as part of the wearable structure 2011 of the wearable band 2010. In some embodiments, sensors 2013 are flush or substantially flush with the wearable structure 2011, such that these sensors do not extend beyond the surface of the wearable structure 2011. Although flush with the wearable structure 2011, sensors 2013 are still configured to contact the user's skin (e.g., via a skin contact surface). Alternatively, in some embodiments, sensors 2013 extend beyond the wearable structure 2011 by a predetermined distance (e.g., 0.1 mm to 2 mm) to contact and press into the user's skin. In some embodiments, sensors 2013 are coupled to an actuator (not shown) configured to adjust the extension height of sensors 2013 (e.g., distance from the surface of the wearable structure 2011) such that sensors 2013 contact and press into the user's skin. In some embodiments, the actuator adjusts the extension height between 0.01 mm and 1.2 mm. This allows users to customize the position of the sensor 2013 to improve the overall comfort of the wearable band 2010 when worn, while still allowing the sensor 2013 to contact the user's skin. In some embodiments, the sensor 2013 is not distinguishable from the wearable structure 2011 when worn by the user.

[0151] The wearable structure 2011 may be formed of an elastic material, elastomer, etc., configured to be stretched and adapted for wear by a user. In some embodiments, the wearable structure 2011 is a textile or woven fabric. As described above, the sensor 2013 may be formed as part of the wearable structure 2011. For example, the sensor 2013 may be molded into the wearable structure 2011, integrated into the woven fabric (e.g., the sensor 2013 may be sewn into the fabric and simulate the flexibility of the fabric and / or may be composed of a series of woven fabric threads).

[0152] Wearable structure 2011 may include sensors 2013, electronic circuits and / or other electronic components (hereinafter referred to as such) to be included in wearable band 2010. Figure 21 (As described) Flexible electronic connectors for interconnection. In some embodiments, the flexible electronic connectors are configured to interconnect sensors 2013, electronic circuitry, and / or other electronic components of the wearable band 2010 with corresponding sensors and / or other electronic components of another electronic device (e.g., the watch body 2020). The flexible electronic connectors are configured to move together with the wearable structure 2011 such that adjustments made to the wearable structure 2011 by the user (e.g., resizing, pulling, folding, etc.) do not cause stress or strain on the electrical coupling of the components of the wearable band 2010.

[0153] As described above, the wearable band 2010 is configured to be worn by a user. Specifically, the wearable band 2010 may be shaped or otherwise manipulated for wear by a user. For example, the wearable band 2010 may be shaped to have a generally circular shape, such that the wearable band may be configured to be worn on the user's forearm or wrist. Alternatively, the wearable band 2010 may be shaped to be worn on another part of the user's body (e.g., the user's upper arm (e.g., around the biceps), forearm, chest, leg, etc.). The wearable band 2010 may include a retaining mechanism 2012 (e.g., a buckle, hook and loop fastener, etc.) for securing the wearable band 2010 to the user's wrist or other body part. When the wearable band 2010 is worn by the user, the sensor 2013 senses data from the user's skin (referred to as sensor data). In some examples, the sensor 2013 of the wearable band 2010 acquires (e.g., senses and records) neuromuscular signals.

[0154] Sensed data (e.g., sensed neuromuscular signals) can be used to detect and / or determine a user's intention to perform certain motor actions. In some examples, sensor 2013 can sense and record neuromuscular signals from the user when the user performs muscle activation (e.g., movement, gesture, etc.). The detected and / or determined motor actions (e.g., phalanges (or fingers) movement, wrist movement, hand movement, and / or other muscle intentions) can be used to determine control commands or control information (instructions to execute certain commands after the data is sensed) for causing the computing device to execute one or more input commands. For example, sensed neuromuscular signals can be used to control certain user interfaces displayed on display 2005 of wrist-worn wearable device 2000, and / or can be sent to a device responsible for rendering an artificial reality environment (e.g., a head-mounted display) to perform actions in the associated artificial reality environment (e.g., to control the movement of a virtual device displayed to the user). Muscle activation performed by a user can include: static gestures, such as placing the user's palm down on a table; dynamic gestures, such as grasping a physical or virtual object; and covert gestures that are imperceptible to another person, such as slightly tensing a joint by coordinating the contraction of opposing muscles or using submuscular activation. Muscle activation performed by a user can also include symbolic gestures (e.g., gestures mapped to other gestures, interactions, or commands based on a gesture vocabulary that specifies a mapping from gestures to commands).

[0155] Sensor data sensed by sensor 2013 can be used to provide users with enhanced interaction with physical objects (e.g., devices communicatively coupled to wearable belt 2010) and / or virtual objects in artificial reality applications generated by artificial reality systems (e.g., user interface objects presented on display 2005 or another computing device (e.g., smartphone)).

[0156] In some embodiments, the wearable band 2010 includes one or more tactile devices 2146 (e.g., vibratory tactile actuators) configured to provide tactile feedback (e.g., skin sensation and / or kinesthetic sensation) to the user's skin. Sensors 2013 and / or tactile devices 2146 (such as...) Figure 21 (As shown) can be configured to run in combination with multiple applications, including but not limited to health monitoring, social media, games and artificial reality (e.g., applications associated with artificial reality).

[0157] The wearable band 2010 may also include a coupling mechanism 2016 for detachably coupling the capsule (e.g., computing unit) or the watch body 2020 to the wearable band 2010 (via a coupling surface of the watch body 2020). For example, the bracket or shape of the coupling mechanism 2016 may correspond to the shape of the watch body 2020 of the wrist wearable device 2000. In particular, the coupling mechanism 2016 may be configured to receive a coupling surface of the watch body 2020 near the bottom side (e.g., the side opposite the front side where the display 2005 of the watch body 2020 is located), allowing a user to push the watch body 2020 down into the coupling mechanism 2016 to attach the watch body 2020 to the coupling mechanism 2016. In some embodiments, the coupling mechanism 2016 may be configured to receive the top side of the watch body 2020 (e.g., the side near the front of the display 2005 on the watch body 2020) which is pushed upward into the bracket rather than downward into the coupling mechanism 2016. In some embodiments, the coupling mechanism 2016 is an integrated component of the wearable strap 2010, such that the wearable strap 2010 and the coupling mechanism 2016 are a single unified structure. In some embodiments, the coupling mechanism 2016 is a frame or housing that allows the coupling surface of the watch body 2020 to be held within or on the coupling mechanism 2016 of the wearable strap 2010 (e.g., bracket, tracking strap, support base, buckle, etc.).

[0158] The coupling mechanism 2016 allows the watch body 2020 to be detachably coupled to the wearable strap 2010 via friction engagement, magnetic coupling, a rotation-based connector, a scissor pin coupling, a retaining spring, one or more magnets, clips, pins, hook-and-loop fasteners, or combinations thereof. A user can perform any type of action to couple the watch body 2020 to the wearable strap 2010 and to detach the watch body 2020 from the wearable strap 2010. For example, a user can twist, slide, rotate, push, pull, or rotate (or combinations thereof) the watch body 2020 relative to the wearable strap 2010 to attach the watch body 2020 to and detach the watch body 2020 from the wearable strap 2010. Alternatively, as discussed below, in some embodiments, the watch body 2020 can be detached from the wearable strap 2010 by actuation of the release mechanism 2029.

[0159] The wearable strap 2010 can be coupled to the watch body 2020 to increase the functionality of the wearable strap 2010 (e.g., converting the wearable strap 2010 into a wrist wearable device 2000, adding additional computing units and / or batteries to increase the computing resources and / or battery life of the wearable strap 2010, adding additional sensors to improve sensed data, etc.). As described above, the wearable strap 2010 and coupling mechanism 2016 are configured to operate independently of the watch body 2020 (e.g., perform functions independently of the watch body). For example, the coupling mechanism 2016 may include one or more sensors 2013 that, with or without the watch body 2020, contact the user's skin when the user wears the wearable strap 2010 and can provide sensor data for determining control commands.

[0160] Users can detach the watch body 2020 from the wearable strap 2010 to reduce the burden of the wrist wearable device 2000 on the user. In embodiments where the watch body 2020 is detachable, the watch body 2020 may be referred to as a detachable structure, such that in these embodiments, the wrist wearable device 2000 includes a wearable portion (e.g., the wearable strap 2010) and a detachable structure (e.g., the watch body 2020).

[0161] Turning to the watch body 2020, in some examples, the watch body 2020 may have a generally rectangular or circular shape. The watch body 2020 is configured to be worn by a user on their wrist or another body part. More specifically, the watch body 2020 is sized for easy carrying by a user, easy attachment to a part of a user's clothing, and / or easy coupling to a wearable strap 2010 (thus forming a wrist wearable device 2000). As described above, the watch body 2020 may have a shape corresponding to the coupling mechanism 2016 of the wearable strap 2010. In some embodiments, the watch body 2020 includes a single release mechanism 2029 or multiple release mechanisms (e.g., two release mechanisms 2029 positioned on opposite sides of the watch body 2020, such as spring-supported buttons) to detach the watch body 2020 from the wearable strap 2010. The release mechanism 2029 may include, but is not limited to, buttons, knobs, plugs, handles, levers, fasteners, buckles, dials, latches, or combinations thereof.

[0162] A user can actuate the release mechanism 2029 by pushing, rotating, lifting, pressing, moving, or performing other actions on it. Actuation of the release mechanism 2029 can release (e.g., detach) the watch body 2020 from the coupling mechanism 2016 of the wearable band 2010, thereby allowing the user to use the watch body 2020 independently of the wearable band 2010, and vice versa. For example, detaching the watch body 2020 from the wearable band 2010 allows the user to use the rear camera 2025b to capture images. Although the release mechanism 2029 is shown positioned at a corner of the watch body 2020, it can be positioned anywhere on the watch body 2020 that is convenient for user actuation. Additionally, in some embodiments, the wearable band 2010 may also include a corresponding release mechanism for detaching the watch body 2020 from the coupling mechanism 2016. In some embodiments, the release mechanism 2029 is optional, and as described above, the body 2020 can be separated from the coupling mechanism 2016 (e.g., by twisting, rotating, etc.).

[0163] The watch body 2020 may include one or more peripheral buttons 2023 and 2027 for performing various operations at the watch body 2020. For example, peripheral buttons 2023 and 2027 may be used to turn on or wake up the display 2005 (e.g., to bring the display 2005 from sleep to active state), unlock the watch body 2020, increase or decrease the volume, increase or decrease the brightness, interact with one or more applications, interact with one or more user interfaces, etc. Additionally or alternatively, in some embodiments, the display 2005 acts as a touchscreen and allows the user to provide one or more inputs for interacting with the watch body 2020.

[0164] In some embodiments, the watch body 2020 includes one or more sensors 2021. The sensors 2021 of the watch body 2020 may be the same as or different from the sensors 2013 of the wearable band 2010. The sensors 2021 of the watch body 2020 may be distributed on the inner and / or outer surfaces of the watch body 2020. In some embodiments, the sensors 2021 are configured to contact the user's skin when the user wears the watch body 2020. For example, the sensors 2021 may be placed on the underside of the watch body 2020, and the coupling mechanism 2016 may be a bracket with an opening that allows the underside of the watch body 2020 to directly contact the user's skin. Alternatively, in some embodiments, the watch body 2020 does not include sensors configured to contact the user's skin (e.g., sensors including those inside and / or outside the watch body 2020, configured to sense data from the watch body 2020 and data from the surrounding environment). In some embodiments, the sensors 2021 are configured to track the position and / or movement of the watch body 2020.

[0165] The watch body 2020 and the wearable band 2010 can share data using wired communication methods (e.g., Universal Asynchronous Receiver / Transmitter (UART), USB transceiver, etc.) and / or wireless communication methods (e.g., Near Field Communication, Bluetooth, etc.). For example, the watch body 2020 and the wearable band 2010 can share data sensed by sensors 2013 and 2021, as well as application-specific and device-specific information (e.g., active and / or available applications, output devices (e.g., display, speaker, etc.), input devices (e.g., touchscreen, microphone, imaging sensor, etc.)).

[0166] In some embodiments, the watch body 2020 may include, but is not limited to, a front-facing camera 2025a and / or a rear-facing camera 2025b, sensors 2021 (e.g., biometric sensors, IMUs, heart rate sensors, oxygen saturation sensors, neuromuscular signal sensors, altimeter sensors, temperature sensors, bioimpedance sensors, pedometer sensors, optical sensors (e.g., imaging sensors 2163), touch sensors, sweat sensors, etc.). In some embodiments, the watch body 2020 may include one or more haptic devices 2176 (e.g., vibratory haptic actuators) configured to provide haptic feedback to the user (e.g., skin sensation and / or kinesthetic sensation, etc.). Sensors 2121 and / or haptic devices 2176 may also be configured to operate in conjunction with multiple applications, including but not limited to health monitoring applications, social media applications, gaming applications, and artificial reality applications (e.g., applications associated with artificial reality).

[0167] As described above, the watch body 2020 and the wearable strap 2010, when coupled, can form a wrist wearable device 2000. The watch body 2020 and the wearable strap 2010, when coupled, can function as a single device to perform the functions (operation, detection, communication, etc.) described herein. In some embodiments, each device may be provided with specific instructions for performing one or more operations of the wrist wearable device 2000. For example, if it is determined that the watch body 2020 does not include a neuromuscular signal sensor, the wearable strap 2010 may include alternative instructions for performing the associated instructions (e.g., providing sensed neuromuscular signal data to the watch body 2020 via different electronic devices). The operations of the wrist wearable device 2000 may be performed by the watch body 2020 alone or by the watch body in conjunction with the wearable strap 2010 (e.g., via a corresponding processor and / or hardware component), or vice versa. In some embodiments, the operations of the wrist wearable device 2000, the watch body 2020, and / or the wearable strap 2010 may be performed in conjunction with one or more processors and / or hardware components.

[0168] For reference below Figure 21 As described in the block diagram, the wearable band 2010 and / or the watch body 2020 may each include independent resources required to perform functions independently. For example, the wearable band 2010 and / or the watch body 2020 may each include a power source (e.g., a battery), memory, data storage, a processor (e.g., a central processing unit (CPU)), communication, a light source, and / or input / output devices.

[0169] Figure 21 Block diagrams are shown of a computing system 2130 corresponding to a wearable strap 2010 and a computing system 2160 corresponding to a watch body 2020, according to some embodiments. According to some embodiments, the computing system 2100 of the wrist wearable device 2000 may include a combination of components of the wearable strap computing system 2130 and components of the watch body computing system 2160.

[0170] The watch body 2020 and / or wearable strap 2010 may include one or more components shown in the watch body computing system 2160. In some embodiments, a single integrated circuit may include all or most of the components of the watch body computing system 2160, which are included in a single integrated circuit. Alternatively, in some embodiments, the components of the watch body computing system 2160 may be included in multiple communication-coupled integrated circuits. In some embodiments, the watch body computing system 2160 may be configured (e.g., via a wired or wireless connection) to couple with the wearable strap computing system 2130, which may allow the two computing systems to share components, distribute tasks, and / or (individually or as a single device) perform other operations described herein.

[0171] The computing system 2160 of the meter may include one or more processors 2179, controllers 2177, peripheral interfaces 2161, power systems 2195, and memory (e.g., memory 2180).

[0172] The power system 2195 may include a charger input 2196, a power-management integrated circuit (PMIC) 2197, and a battery 2198. In some embodiments, the watch body 2020 and the wearable strap 2010 may have their own batteries (e.g., batteries 2198 and 2159) and may share power with each other. The watch body 2020 and the wearable strap 2010 may use various technologies to receive charge. In some embodiments, the watch body 2020 and the wearable strap 2010 may use wired charging components (e.g., a power cord) to receive charge. Alternatively or additionally, the watch body 2020 and / or the wearable strap 2010 may be configured for wireless charging. For example, a portable charging device may be designed to mate with a portion of the watch body 2020 and / or a portion of the wearable strap 2010 and wirelessly deliver available power to the battery 2198 of the watch body 2020 and / or the battery 2159 of the wearable strap 2010. The watch body 2020 and the wearable band 2010 may have independent power systems (e.g., power systems 2195 and 2156, respectively) to enable each to operate independently. The watch body 2020 and the wearable band 2010 may also share power (e.g., one can charge the other) via their respective PMICs (e.g., PMICs 2197 and 2158) and charger inputs (e.g., charger inputs 2157 and 2196), which can share power via power conductors and ground conductors and / or via wireless charging antennas.

[0173] In some embodiments, the peripheral interface 2161 may include one or more sensors 2121. Sensor 2121 may include one or more coupling sensors 2162 for detecting when the watch body 2020 is coupled to another electronic device (e.g., the wearable band 2010). Sensor 2121 may include one or more imaging sensors 2163 (e.g., one or more of a camera 2125 and / or a separate imaging sensor 2163 (e.g., a thermal imaging sensor)). In some embodiments, sensor 2121 may include one or more SpO2 sensors 2164. In some embodiments, sensor 2121 may include one or more bioelectric potential signal sensors (e.g., an EMG sensor 2165, which may be disposed on the user-facing internal portion of the watch body 2020 and / or the wearable band 2010). In some embodiments, sensor 2121 may include one or more capacitive sensors 2166. In some embodiments, sensor 2121 may include one or more heart rate sensors 2167. In some embodiments, sensor 2121 may include one or more IMU sensors 2168. In some embodiments, one or more IMU sensors 2168 may be configured to detect the movement of a user’s hand, or the movement of the watch body 2020 in other positions where it is placed or held.

[0174] In some embodiments, one or more of the sensors 2121 may provide an example human-machine interface. For example, a set of neuromuscular sensors (e.g., EMG sensor 2165) may be arranged circumferentially along the wearable band 2010, wherein the inner surface of the EMG sensor 2165 is configured to contact the user's skin. Any suitable number of neuromuscular sensors may be used (e.g., 2 to 20 sensors). The number and arrangement of the neuromuscular sensors may depend on the specific application for which the wearable device is used. For example, the wearable band 2010 may be used to generate control information for controlling an augmented reality system, controlling a robot, controlling a vehicle, scrolling text, controlling a virtual avatar, or controlling any other suitable control task.

[0175] In some embodiments, the neuromuscular sensors may be coupled together using flexible electronics incorporated into a wireless device, and the outputs of one or more sensing elements may optionally be processed using hardware signal processing circuitry (e.g., to perform amplification, filtering, and / or rectification). In other embodiments, at least some of the signal processing of the outputs of the sensing elements may be performed in software (e.g., processor 2179). Therefore, signal processing of the signals sampled by the sensors may be performed in hardware, in software, or by any suitable combination of hardware and software, as the aspects of the techniques described herein are not limited in this respect.

[0176] Neuromuscular signals can be processed in various ways. For example, the output of the EMG sensor 2165 can be provided to an analog front end, which can be configured to perform analog processing (e.g., amplification, noise reduction, filtering, etc.) on the recorded signal. The processed analog signal can then be provided to an analog-to-digital converter, which can convert the analog signal into a digital signal that can be processed by one or more computer processors. Furthermore, although this example is discussed in the context of interfacing with an EMG sensor, the embodiments described herein can also be implemented in wearable interfaces with other types of sensors, including but not limited to mechanomyography (MMG) sensors, sonomyography (SMG) sensors, and electrical impedance tomography (EIT) sensors.

[0177] In some embodiments, the peripheral interface 2161 includes a Near Field Communication (NFC) component 2169, a Global Positioning System (GPS) component 2170, a Long-Term Evolution (LTE) component 2171, and / or a Wi-Fi and / or Bluetooth communication component 2172. In some embodiments, the peripheral interface 2161 includes one or more buttons 2173 (e.g., Figure 20 The peripheral interface 2161 includes peripheral buttons 2023 and 2027, which, when selected by the user, cause an operation to be performed at the body 2020. In some embodiments, the peripheral interface 2161 includes one or more indicators (e.g., light-emitting diodes, LEDs) to provide the user with visual indicators (e.g., received message, low battery, active microphone and / or camera, etc.).

[0178] The watch body 2020 may include at least one display 2005 for displaying a visual representation of information or data to a user, including user interface elements and / or three-dimensional virtual objects. The display may also include a touchscreen for inputting user input (e.g., touch gestures and swipe gestures). The watch body 2020 may include at least one speaker 2174 and at least one microphone 2175 for providing audio signals to the user and receiving audio input from the user. The user can provide user input through the microphone 2175 and can also receive audio output from the speaker 2174 as part of a haptic event provided by a haptic controller 2178. The watch body 2020 may include at least one camera 2125, including a front-facing camera 2125a and a rear-facing camera 2125b. The camera 2125 may include an ultra-wide-angle camera, a wide-angle camera, a fisheye camera, a spherical camera, a telephoto camera, a depth-sensing camera, or other types of cameras.

[0179] The watch body computing system 2160 may include one or more haptic controllers 2178 and associated components (e.g., haptic devices 2176) for providing haptic events at the watch body 2020 (e.g., a vibrational sensation or audio output responding to an event at the watch body 2020). The haptic controllers 2178 may communicate with one or more haptic devices 2176 (e.g., electroacoustic devices), including speakers in one or more loudspeakers 2174 and / or other audio components and / or electromechanical devices that convert energy into linear motion (e.g., motors, electromagnetic coils, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert electrical signals into haptic outputs on the device)). The haptic controllers 2178 can provide haptic events that a user of the watch body 2020 can feel. In some embodiments, the one or more haptic controllers 2178 may receive input signals from an application in application 2182.

[0180] In some embodiments, the table body computing system 2160 may include a memory 2180, which may be controlled by one or more memory controllers of the controller 2177. In some embodiments, software components stored in the memory 2180 include one or more applications 2182 configured to perform operations at the table body 2020. In some embodiments, the one or more applications 2182 may include games, word processors, messaging applications, calling applications, web browsers, social media applications, media streaming applications, financial applications, calendars, clocks, etc. In some embodiments, software components stored in the memory 2180 include one or more communication interface modules 2183 as defined above. In some embodiments, software components stored in the memory 2180 include: one or more graphics modules 2184 for rendering, encoding, and / or decoding audio data and / or video data; and one or more data management modules 2185 for collecting, organizing, and / or providing access to data 2187 stored in the memory 2180. In some embodiments, one or more applications and / or one or more modules in the applications 2182 may work together to perform various tasks at the table body 2020.

[0181] In some embodiments, the software components stored in memory 2180 may include one or more operating systems 2181 (e.g., a Linux-based operating system, an Android operating system, etc.). Memory 2180 may also include data 2187. Data 2187 may include data 2188A, sensor data 2189A, media content data 2190, and application data 2191.

[0182] It should be understood that the table body computing system 2160 is an example of a computing system within the table body 2020, and the table body 2020 may have more or fewer components than those shown in the table body computing system 2160, may combine two or more components, and / or may have different configurations and / or arrangements of these components. The various components shown in the table body computing system 2160 are implemented in hardware, software, firmware, or combinations thereof (including one or more signal processing circuits and / or application-specific integrated circuits).

[0183] Turning to wearable band computing system 2130, one or more components that may be included in wearable band 2010 are shown. Wearable band computing system 2130 may include more or fewer components than those shown in body computing system 2160, may combine two or more components, and / or may have different configurations and / or arrangements of some or all of these components. In some embodiments, all or most of the components of wearable band computing system 2130 are included in a single integrated circuit. Alternatively, in some embodiments, the components of wearable band computing system 2130 are included in multiple communication-coupled integrated circuits. As described above, in some embodiments, wearable band computing system 2130 is configured to couple with body computing system 2160 (e.g., via a wired or wireless connection), which allows the two computing systems to share components, assign tasks, and / or (individually or as a single device) perform other operations described herein.

[0184] Similar to the wearable computing system 2160, the wearable computing system 2130 may include: one or more processors 2149; one or more controllers 2147 (including one or more haptic controllers 2148); a peripheral interface 2131, which may include one or more sensors 2113 and other peripheral devices; a power supply (e.g., a power system 2156); and a memory (e.g., a memory 2150), which includes an operating system (e.g., an operating system 2151), data (e.g., data 2154, which includes data 2188B, sensor data 2189B, etc.) and one or more modules (e.g., a communication interface module 2152, a data management module 2153, etc.).

[0185] One or more of the sensors 2113 may be similar to sensor 2121 of the body computing system 2160. For example, sensor 2113 may include one or more coupling sensors 2132, one or more SpO2 sensors 2134, one or more EMG sensors 2135, one or more capacitive sensors 2136, one or more heart rate sensors 2137, and one or more IMU sensors 2138.

[0186] Peripheral interface 2131 may also include other components similar to those included in peripheral interface 2161 of watch computing system 2160, as described above with reference to peripheral interface 2161. These other components include NFC component 2139, GPS component 2140, LTE component 2141, Wi-Fi and / or Bluetooth communication component 2142, and / or one or more haptic devices 2146. In some embodiments, peripheral interface 2131 includes one or more buttons 2143, a display 2133, a speaker 2144, a microphone (MIC) 2145, and a camera 2155. In some embodiments, peripheral interface 2131 includes one or more indicators, such as LEDs.

[0187] It should be understood that the wearable band computing system 2130 is an example of a computing system within the wearable band 2010, and the wearable band 2010 may have more or fewer components than those shown in the wearable band computing system 2130, may combine two or more components, and / or may have different configurations and / or arrangements of these components. The various components shown in the wearable band computing system 2130 may be implemented as one or more of a combination of hardware, software, or firmware (including one or more signal processing circuits and / or application-specific integrated circuits).

[0188] refer to Figure 20 The wrist wearable device 2000 is an example of a wearable strap 2010 and a watch body 2020 coupled together, and therefore the wrist wearable device 2000 will be understood to include the components shown and described for the wearable strap computing system 2130 and the watch body computing system 2160. In some embodiments, the wrist wearable device 2000 has a split architecture (e.g., a split mechanical architecture, a split electronic architecture, etc.) between the watch body 2020 and the wearable strap 2010. In other words, all the components shown in the wearable strap computing system 2130 and the watch body computing system 2160 can be accommodated or otherwise arranged in the combined wrist wearable device 2000, or accommodated or otherwise arranged within a single component in the watch body 2020, the wearable strap 2010 and / or portions thereof (e.g., the coupling mechanism 2016 of the wearable strap 2010).

[0189] The above technology can be used with any device for sensing neuromuscular signals, but it can also be used with other types of wearable devices for sensing neuromuscular signals, such as body wearables or head wearables that may have neuromuscular sensors closer to the brain or spine.

[0190] In some embodiments, the wrist wearable device 2000 may be used in conjunction with a head wearable device (e.g., AR glasses 2200 and VR system 2310) and / or HIPD, and the wrist wearable device 2000 may also be configured to allow a user to control any aspect of the artificial reality (e.g., by controlling user interface objects in the artificial reality using EMG-based gestures, and / or by allowing a user to interact with a touchscreen on the wrist wearable device to also control aspects of the artificial reality). Having described example wrist wearable devices in this manner, attention now turns to example head wearable devices, such as AR glasses 2200 and VR headset 2310.

[0191] Figures 22 to 24 An example artificial reality system is shown, which can be used as or in conjunction with a wrist-worn wearable device 2000. In some embodiments, such as Figure 22 As shown, the AR system 2200 includes glasses device 2202. In some embodiments, such as Figure 23A and Figure 23B As shown, VR system 2310 includes a head-mounted display (HMD) 2312. In some embodiments, AR system 2200 and VR system 2310 may include one or more similar components (e.g., components for presenting an interactive artificial reality environment, such as a processor, memory, and / or presentation device, including one or more displays and / or one or more waveguides), some of which will be referenced to Figure 24 A more detailed description is provided below. As described herein, a head-mounted wearable device may include components of glasses device 2202 and / or head-mounted display 2312. Some embodiments of the head-mounted wearable device do not include any display, including any of the displays described with reference to AR system 2200 and / or VR system 2310. Although the example artificial reality systems are described herein as AR system 2200 and VR system 2310, respectively, one or both of the example AR systems described herein may be configured to present a fully immersive virtual reality scene presented in substantially the entire field of view of the user, or a more subtle augmented reality scene presented in a portion (not the entirety) of the user's field of view.

[0192] Figure 22 An example visual depiction of an AR system 2200 including glasses device 2202 (e.g., herein the AR system 2200 may also be described as augmented reality glasses and / or smart glasses) is shown. The AR system 2200 may include… Figure 22Additional electronic components (e.g., wearable accessory devices and / or intermediate processing devices, not shown) that communicate electronically with or are otherwise configured to be used in conjunction with the eyewear device 2202. In some embodiments, the wearable accessory device and / or intermediate processing device may be configured to be connected via a coupling sensor 2424 ( Figure 24 The electronic communication coupling mechanism is coupled to the eyewear device 2202, wherein the coupling sensor 2424 can detect when the electronic device is physically or electronically coupled to the eyewear device 2202. In some embodiments, the eyewear device 2202 may be configured to couple to the housing 2490. Figure 24 The housing may include one or more additional coupling mechanisms configured to couple with additional accessory devices. Figure 22 The components shown can be implemented in hardware, software, firmware, or a combination thereof (including one or more signal processing components and / or application-specific integrated circuits (ASICs)).

[0193] The eyewear device 2202 includes mechanical eyewear components, including a frame 2204 configured to hold one or more lenses (e.g., one or both of lenses 2206-1 and 2206-2). Those skilled in the art will recognize that the eyewear device 2202 may include additional mechanical components, such as hinges configured to allow partial folding and unfolding of the frame 2204 of the eyewear device 2202, a bridge configured to span the gap between lenses 2206-1 and 2206-2 and rest on the user's nose, nose pads configured to rest on the bridge of the nose and provide support for the eyewear device 2202, earpieces configured to rest on the user's ears and provide additional support for the eyewear device 2202, and temples configured to extend from the hinges to the earpieces of the eyewear device 2202, etc. Those skilled in the art will also recognize that some examples of the AR system 2200 may not include the mechanical components described herein. For example, smart contact lenses configured to present artificial reality to users may not include any components of the eyewear device 2202.

[0194] Eyeglasses device 2202 includes multiple electronic components, many of which will be described in the following reference. Figure 24 To provide a more detailed description. Figure 22The diagram illustrates some example electronic components, including acoustic sensors 2225-1, 2225-2, 2225-3, 2225-4, 2225-5, and 2225-6, which may be distributed along a large portion of the frame 2204 of the eyewear device 2202. The eyewear device 2202 also includes a left camera 2239A and a right camera 2239B located on different sides of the frame 2204. The eyewear device 2202 also includes a processor 2248 (or any other suitable type or form of integrated circuit) embedded in a portion of the frame 2204.

[0195] Figure 23A and Figure 23B A VR system 2310 according to some embodiments is shown, which includes a head-mounted display (HMD) 2312 (e.g., also referred to herein as an artificial reality head-mounted device, a head-worn device, a VR head-mounted device, etc.). As mentioned, some artificial reality systems (e.g., AR system 2200) may substantially use virtual experiences to replace one or more of a user's visual and / or other sensory perceptions of the real world, rather than blending artificial reality with real reality.

[0196] The HMD 2312 includes a front body 2314 and a frame 2316 (e.g., a strip or strap) shaped to fit the user's head. In some embodiments, the front body 2314 and / or frame 2316 include one or more electronic components (e.g., a display, IMU, tracking transmitter, or detector) for facilitating the presentation and / or interaction with AR and / or VR systems. In some embodiments, such as Figure 23B As shown, the HMD 2312 includes an output audio transducer (e.g., audio transducer 2318). In some embodiments, such as Figure 23B As shown, one or more components (e.g., one or more output audio transducers 2318 and frame 2316) (e.g., a portion or all of frame 2316 and / or audio transducers 2318) may be configured to be attached to and detached from HMD 2312 (e.g., detachably attached to HMD 2312). In some embodiments, coupling a detachable component to HMD 2312 enables the detachable component to enter into electronic communication with HMD 2312.

[0197] Figure 23A and Figure 23BThe VR system 2310 is also shown to include one or more cameras, such as a left camera 2339A and a right camera 2339B, which may resemble the left and right cameras 2239A and 2239B on the frame 2204 of the glasses device 2202. In some embodiments, the VR system 2310 includes one or more additional cameras (e.g., cameras 2339C and 2339D) that can be configured to enhance the image data acquired by the left camera 2339A and right camera 2339B by providing more information. For example, camera 2339C may be used to provide color information not identified by cameras 2339A and 2339B. In some embodiments, one or more of the cameras 2339A through 2339D may include an optional infrared (IR) cutoff filter configured to remove IR light received at the respective camera sensor.

[0198] Figure 24 A computing system 2420 and an optional housing 2490 are shown, both of which display components that may be included in the AR system 2200 and / or the VR system 2310. In some embodiments, the optional housing 2490 may include more or fewer components, depending on the practical limitations of the respective AR system described.

[0199] In some embodiments, the computing system 2420 may include one or more peripheral interfaces 2422A, and / or an optional housing 2490 may include one or more peripheral interfaces 2422B. Each of the computing system 2420 and the optional housing 2490 may also include one or more power systems 2442A and 2442B, one or more controllers 2446 (including one or more haptic controllers 2447), one or more processors 2448A and 2448B (as defined above, including any examples provided), and memories 2450A and 2450B, all of which may communicate electronically with each other. For example, the one or more processors 2448A and 2448B may be configured to execute instructions stored in memories 2450A and 2450B, which may cause a controller in one or more controllers 2446 to cause operation to be performed at one or more peripheral devices connected to peripheral interfaces 2422A and / or 2422B. In some embodiments, each of the described operations may be powered by power provided by power systems 2442A and / or 2442B.

[0200] In some embodiments, peripheral interface 2422A may include one or more devices configured as part of computing system 2420, some of which have been defined and / or referenced above. Figure 20 and Figure 21 The wrist-worn wearable device shown is described. For example, peripheral interface 2422A may include one or more sensors 2423A. Some example sensors 2423A include one or more coupling sensors 2424, one or more acoustic sensors 2425, one or more imaging sensors 2426, one or more EMG sensors 2427, one or more capacitive sensors 2428, one or more IMU sensors 2429, and / or any other type of sensor explained above or described with reference to any other embodiments discussed herein.

[0201] In some embodiments, peripheral interfaces 2422A and 2422B may include one or more additional peripheral devices, including one or more NFC devices 2430, one or more GPS devices 2431, one or more LTE devices 2432, one or more Wi-Fi and / or Bluetooth devices 2433, one or more buttons 2434 (e.g., including slide-on or otherwise adjustable buttons), one or more displays 2435A and 2435B, one or more speakers 2436A and 2436B, one or more microphones 2437, one or more cameras 2438A and 2438B (e.g., including left camera 2439A and / or right camera 2439B), one or more haptic devices 2440, and / or any other type of peripheral device as defined above or described with reference to any other embodiments discussed herein.

[0202] AR systems can include various types of visual feedback mechanisms (e.g., presentation devices). For example, the display devices in AR system 2200 and / or VR system 2310 can include one or more liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic LED (OLED) displays, and / or any other suitable type of display screen. An AR system can include a single display screen (e.g., configured for binocular viewing), or can provide separate displays for each eye, which can provide additional flexibility for zoom adjustment and / or correction of refractive errors associated with the user's vision. Some embodiments of AR systems also include an optical subsystem with one or more lenses (e.g., conventional concave or convex lenses, Fresnel lenses, or adjustable liquid lenses) through which the user views the display screen.

[0203] For example, corresponding displays 2435A and 2435B may be coupled to each of lenses 2206-1 and 2206-2 of the AR system 2200. Displays 2435A and 2435B may be coupled to each of lenses 2206-1 and 2206-2, and they may present an image or a series of images to the user together or independently. In some embodiments, the AR system 2200 includes a single display 2435A or 2435B (e.g., a near-eye display) or two or more displays 2435A and 2435B. In some embodiments, a first group of one or more displays 2435A and 2435B may be used to present an augmented reality environment, and a second group of one or more display devices 2435A and 2435B may be used to present a virtual reality environment. In some embodiments, one or more waveguides are used in conjunction with presenting artificial reality content to the user of the AR system 2200 (e.g., as a means of delivering light from one or more displays 2435A and 2435B to the user's eyes). In some embodiments, one or more waveguides are wholly or partially integrated into the eyewear device 2202. In addition to or as an alternative to a display screen, some artificial reality systems include one or more projection systems. For example, each display device in AR system 2200 and / or VR system 2310 may include (e.g., using waveguides) a micro-LED projector that projects light onto the display device, such as a transparent combination lens that allows ambient light to pass through. The display device can refract the projected light toward the user's pupil, allowing the user to simultaneously view both artificial reality content and the real world. Artificial reality systems can also be configured with any other suitable type or form of image projection system. In some embodiments, one or more waveguides are provided in addition to or as an alternative to one or more displays 2435A and 2435B.

[0204] The computing system 2420 of the AR system 2200 or the optional housing 2490 of the VR system 2310 may include some or all of the components of the power systems 2442A and 2442B. The power systems 2442A and 2442B may include one or more charger inputs 2443, one or more PMICs 2444, and / or one or more batteries 2445A and 2444B.

[0205] Memory 2450A and 2450B may include instructions and data, some or all of which may be stored within memory 2450A and 2450B as a non-transitory computer-readable storage medium. For example, memory 2450A and 2450B may include one or more operating systems 2451, one or more applications 2452, one or more communication interface applications 2453A and 2453B, one or more graphics applications 2454A and 2454B, one or more AR processing applications 2455A and 2455B, and / or any other type of data as defined above or described with reference to any other embodiments discussed herein.

[0206] Memory 2450A and 2450B also include data 2460A and 2460B, which can be used in conjunction with one or more of the applications discussed above. Data 2460A and 2460B may include data 2461, sensor data 2462A and 2462B, media content data 2463A, AR application data 2464A and 2464B, and / or any other type of data as defined above or described with reference to any other embodiments discussed herein.

[0207] In some embodiments, the controller 2446 of the glasses device 2202 can process information generated by sensors 2423A and / or 2423B on the glasses device 2202, and / or by another electronic device within the AR system 2200. For example, the controller 2446 can process information from acoustic sensors 2225-1 and 2225-2. For each detected sound, the controller 2446 can perform direction of arrival (DOA) estimation to estimate the direction from which the detected sound arrives at the glasses device 2202 of the AR system 2200. When one or more of the acoustic sensors 2425 (e.g., acoustic sensors 2225-1, 2225-2) detect sound, the controller 2446 can use this information to populate the audio dataset (e.g., in...). Figure 24 (represented as sensor data 2462A and 2462B).

[0208] In some embodiments, physical electronic connectors can transmit information between the eyewear device 2202 and another electronic device, and / or between one or more processors 2248, 2448A, 2448B and controller 2446 in the AR system 2200 or VR system 2310. This information can be in the form of optical data, electronic data, wireless data, or any other transmissible data format. Delegating the processing of information generated by the eyewear device 2202 to an intermediate processing device can reduce the weight and heat of the eyewear device, making it more comfortable and safer for the user. In some embodiments, an optional wearable accessory device (e.g., an electronic neckband) is coupled to the eyewear device 2202 via one or more connectors. The connectors can be wired or wireless and can include electronic components and / or non-electronic (e.g., structural) components. In some embodiments, the eyewear device 2202 and the wearable accessory device can operate independently without any wired or wireless connection between them.

[0209] In some cases, pairing an external device (e.g., a mid-processing device (e.g., HIPD 1606, 1706, 1806)) with glasses device 2202 (e.g., as part of AR system 2200) enables glasses device 2202 to achieve a form factor similar to that of a pair of glasses while still providing sufficient battery power and computing power for the extended capabilities. Some or all of the battery power, computing resources, and / or additional features of AR system 2200 may be provided by the paired device or shared between the paired device and glasses device 2202, thereby reducing the overall weight, heat profile, and form factor of glasses device 2202 while allowing glasses device 2202 to maintain its desired functionality. For example, wearable accessory devices may allow components otherwise included on glasses device 2202 to be included in the wearable accessory device and / or mid-processing device, thereby transferring the weight load from the user's head and neck to one or more other parts of the user's body. In some embodiments, the mid-processing device has a larger surface area thereon that diffuses and disperses heat to the surrounding environment. Therefore, the intermediate processing device allows for greater battery capacity and computing power compared to what might otherwise be available on a standalone glasses device 2202. Since the weight carried by the wearable accessory device is less intrusive to the user than the weight carried by the glasses device 2202, users can tolerate wearing a lighter glasses device and carrying or wearing paired devices for longer periods compared to enduring the weight of a standalone glasses device, thus allowing the artificial reality environment to be more fully integrated into the user's daily activities.

[0210] AR systems can include various types of computer vision components and computer vision subsystems. For example, AR system 2200 and / or VR system 2310 can include one or more optical sensors, such as two-dimensional (2D) cameras or three-dimensional (3D) cameras, time-of-flight depth sensors, structured light emitters and detectors, single-beam rangefinders or scanning laser rangefinders, 3D LiDAR sensors, and / or any other suitable type or form of optical sensor. AR systems can process data from one or more of these sensors to identify the user's location and / or aspects of the user's real-world physical environment, including the location of real-world objects within the real-world physical environment. In some embodiments, the methods described herein are used to map the real world to provide the user with context about their surroundings, and / or to generate digital twins (e.g., interactive virtual objects), and various other functionalities. For example, Figure 23A and Figure 23B A VR system 2310 with cameras 2339A to 2339D is shown. This VR system can be used to provide depth information to create voxel fields and two-dimensional meshes, thereby providing the user with object information to avoid collisions.

[0211] In some embodiments, AR system 2200 and / or VR system 2310 may include a haptic feedback system that can be integrated into headwear, gloves, bodysuits, handheld controllers, environmental devices (e.g., chairs or mats), and / or any other type of device or system (e.g., wearable devices discussed herein). The haptic feedback system can provide various types of skin feedback, including vibration, force, tension, shear stress, texture, and / or temperature. The haptic feedback system can also provide various types of kinematic feedback, such as motion and compliance. Haptic feedback can be implemented using motors, piezoelectric actuators, fluid systems, and / or various other types of feedback mechanisms. The haptic feedback system can be implemented independently of other artificial reality devices, within other artificial reality devices, and / or in conjunction with other artificial reality devices.

[0212] In some embodiments of artificial reality systems (e.g., AR system 2200 and / or VR system 2310), ambient light (e.g., a real-time feed of the user's surroundings as normally perceived) may penetrate the display elements of the corresponding head-mounted wearable device that is presenting aspects of the AR system. In some embodiments, ambient light may penetrate a portion (less than the entire AR environment) of the AR environment presented within the user's field of view (e.g., a portion of the AR environment where physical objects are located within a designated boundary (e.g., a guard boundary) in the user's real-world environment, which is configured to be used by the user while interacting with the AR environment). For example, visual user interface elements (e.g., notification user interface elements) may be presented on the head-mounted wearable device, and a certain amount of ambient light (e.g., 15% to 50% of ambient light) may penetrate the user interface element, allowing the user to distinguish at least a portion of the physical environment on which the user interface element is displayed.

[0213] In some embodiments, the system described herein may also include an eye-tracking subsystem designed to identify and track various characteristics of one or both eyes of a user (e.g., the user's gaze direction). In some examples, the phrase "eye tracking" may refer to the process of measuring, detecting, sensing, determining, and / or monitoring the position, orientation, and / or movement of the eyes. The disclosed system may measure the position, orientation, and / or movement of the eyes in a variety of different ways, including by using various optical-based eye-tracking techniques, ultrasonic-based eye-tracking techniques, etc. The eye-tracking subsystem may be configured in a variety of different ways and may include a variety of different eye-tracking hardware components or other computer vision components. For example, the eye-tracking subsystem may include a variety of different optical sensors, such as two-dimensional (2D) cameras or 3D cameras, time-of-flight depth sensors, single-beam rangefinders or scanning laser rangefinders, 3D LiDAR sensors, and / or any other suitable type or form of optical sensor. In this example, the processing subsystem can process data from one or more of these sensors to measure, detect, determine, and / or otherwise monitor the position, orientation, and / or movement of a user's single or binocular eye.

[0214] Figure 25 This is an illustration of an example system 2500 that incorporates an eye-tracking subsystem capable of tracking a user's single or binocular eyes. (See illustration for example.) Figure 25As depicted, system 2500 may include a light source 2502, an optical subsystem 2504, an eye-tracking subsystem 2506, and / or a control subsystem 2508. In some examples, light source 2502 may generate light to produce an image (e.g., to be presented to the viewer's eye 2501). Light source 2502 may represent any of a variety of suitable devices. For example, light source 2502 may include a two-dimensional projector (e.g., an LCoS display), a scanning source (e.g., a scanning laser), or other devices (e.g., an LCD, LED display, OLED display, active-matrix OLED (AMOLED) display, transparent OLED (TOLED) display, waveguide, or some other display capable of generating light to present an image to a viewer). In some examples, the image may represent a virtual image, which may refer to an optical image formed according to the apparent divergence of light rays from a point in space, rather than an image formed according to the actual divergence of light rays.

[0215] In some embodiments, the optical subsystem 2504 may receive light generated by the light source 2502 and generate a converging beam 2520 including an image based on the received light. In some examples, the optical subsystem 2504 may include any number of lenses (e.g., Fresnel lenses, convex lenses, concave lenses), apertures, filters, mirrors, prisms, and / or other optical components that may be combined with actuators and / or other devices. In particular, the actuators and / or other devices may translate and / or rotate one or more of the plurality of optical components to change one or more aspects of the converging beam 2520. Furthermore, various mechanical couplings may be used to maintain the relative spacing and / or orientation of the respective optical components in any suitable combination.

[0216] In one embodiment, the eye-tracking subsystem 2506 can generate tracking information indicating the gaze angle of a viewer's eye 2501. In this embodiment, the control subsystem 2508 can control aspects of the optical subsystem 2504 (e.g., the angle of incidence of the converging beam 2520) based at least in part on this tracking information. Additionally, in some examples, the control subsystem 2508 can store and utilize historical tracking information (e.g., a history of tracking information over a given duration (e.g., a fraction of a second)) to predict the gaze angle of the eye 2501 (e.g., the angle between the visual axis and the anatomical axis of the eye 2501). In some embodiments, the eye-tracking subsystem 2506 can detect radiation emitted from a part of the eye 2501 (e.g., the cornea, iris, or pupil, etc.) to determine the current gaze angle of the eye 2501. In other examples, the eye-tracking subsystem 2506 can employ a wavefront sensor to track the current position of the pupil.

[0217] Any number of techniques can be used to track the eye 2501. Some techniques may involve illuminating the eye 2501 with infrared light and measuring the reflection using at least one optical sensor tuned to be sensitive to infrared light. Information about how infrared light is reflected from the eye 2501 can be analyzed to determine one or more locations, one or more orientations, and / or one or more movements of one or more eye features (e.g., cornea, pupil, iris, and / or retinal vessels).

[0218] In some examples, the radiation acquired by the sensors of the eye-tracking subsystem 2506 can be digitized (i.e., converted into an electronic signal). Furthermore, the sensors can send this digital representation of the electronic signal to one or more processors (e.g., a processor associated with a device including the eye-tracking subsystem 2506). The eye-tracking subsystem 2506 can include any of a variety of sensors in various different configurations. For example, the eye-tracking subsystem 2506 can include an infrared detector that responds to infrared radiation. The infrared detector can be a thermal detector, a photon detector, and / or any other suitable type of detector. The thermal detector can include a detector that responds to the thermal effects of incident infrared radiation.

[0219] In some examples, one or more processors may process digital representations generated by one or more sensors of the eye-tracking subsystem 2506 to track the movement of the eye 2501. In another example, these processors may track the movement of the eye 2501 by executing an algorithm represented by computer-executable instructions stored in non-transitory memory. In some examples, on-chip logic (e.g., an application-specific integrated circuit or ASIC) may be used to execute at least a portion of such an algorithm. As noted, the eye-tracking subsystem 2506 may be programmed to track the movement of the eye 2501 using the output of one or more sensors. In some embodiments, the eye-tracking subsystem 2506 may analyze the digital representations generated by the sensors to extract eye rotation information from changes in reflection. In one embodiment, the eye-tracking subsystem 2506 may use corneal reflections or flashes (also known as Purkinje images) and / or the center of the pupil 2522 of the eye as features to be tracked over time.

[0220] In some embodiments, the eye-tracking subsystem 2506 can generate a corneal reflection using the center of the pupil 2522 of the eye and uncollimated infrared or near-infrared light. In these embodiments, the eye-tracking subsystem 2506 can calculate the gaze direction of the eye 2501 using the vector between the center of the pupil 2522 and the corneal reflection. In some embodiments, the disclosed system can perform a calibration process on an individual (using, for example, supervised or unsupervised techniques) before tracking the user's eyes. For example, the calibration process may include guiding the user to look at one or more points displayed on a display while the eye-tracking system records values ​​corresponding to each gaze position associated with each point.

[0221] In some embodiments, the eye-tracking subsystem 2506 may use two types of infrared and / or near-infrared (also known as active light) eye-tracking techniques: bright-pupil eye-tracking and dark-pupil eye-tracking, which can be distinguished based on the position of the illumination source relative to the optical elements used. If the illumination is coaxial with the light path, the eye 2501 can act as a retroreflector when light is reflected back from the retina, producing a bright-pupil effect similar to the red-eye effect in photography. If the illumination source is offset from the light path, the pupil 2522 of the eye may appear darker because the retroreflection from the retina is deviated from the sensor orientation. In some embodiments, bright-pupil tracking can produce greater iris / pupil contrast to allow robust eye-tracking in the presence of iris pigmentation and can be characterized by reduced interference (e.g., interference from eyelashes and other blurry features). Bright-pupil tracking can also allow tracking in illumination conditions ranging from complete darkness to very bright environments.

[0222] In some embodiments, the control subsystem 2508 may control the light source 2502 and / or the optical subsystem 2504 to reduce optical aberrations (e.g., chromatic aberration and / or monochromatic aberration) in the image that may be caused by or affected by the eye 2501. In some examples, as mentioned above, the control subsystem 2508 may use tracking information from the eye-tracking subsystem 2506 to perform this control. For example, when controlling the light source 2502, the control subsystem 2508 may (e.g., through image rendering) modify the light generated by the light source 2502 to modify (e.g., pre-distort) the image, thereby reducing aberrations in the image caused by the eye 2501.

[0223] The disclosed system can track both the position and relative size of the pupil (e.g., due to pupil dilation and / or constriction). In some examples, the eye-tracking devices and components (e.g., sensors and / or sources) used to detect and / or track the pupil may be different (or differently calibrated) for different types of eyes. For example, the frequency range of the sensors may be different (or individually calibrated) for eyes of different colors and / or different pupil types, and / or sizes, etc. Therefore, it may be necessary to calibrate the various eye-tracking components (e.g., infrared sources and / or sensors) described herein for each individual user and / or eye.

[0224] The disclosed system can track eyes with and without ophthalmic correction (e.g., correction provided by contact lenses worn by the user). In some embodiments, ophthalmic corrective elements (e.g., adjustable lenses) can be directly integrated into the artificial reality system described herein. In some examples, the color of the user's eyes may require modification of the corresponding eye-tracking algorithm. For example, the eye-tracking algorithm may need to be modified at least in part based on, for example, the different color contrast between brown and blue eyes.

[0225] Figure 26 yes Figure 25 A more detailed illustration of various aspects of the eye-tracking subsystem shown is provided below. As illustrated, the eye-tracking subsystem 2600 may include at least one source 2604 and at least one sensor 2606. Source 2604 generally represents an element capable of emitting radiation of any type or form. In one example, source 2604 may generate visible radiation, infrared radiation, and / or near-infrared radiation. In some examples, source 2604 may radiate the non-collimated infrared and / or near-infrared portions of the electromagnetic spectrum toward the user's eye 2602. Source 2604 may utilize various sampling rates and speeds. For example, the disclosed system may use a source with a high sampling rate to acquire fixational eye movements of the user's eye 2602 and / or accurately measure the saccade dynamics of the user's eye 2602. As mentioned above, any type or form of eye-tracking technology (including optical eye-tracking technology, ultrasound-based eye-tracking technology, etc.) can be used to track the user's eye 2602.

[0226] Sensor 2606 broadly represents any type or form of element capable of detecting radiation (e.g., radiation reflected from a user's eye 2602). Examples of sensor 2606 include, but are not limited to, charge-coupled devices (CCDs), photodiode arrays, and / or sensor devices based on complementary metal-oxide-semiconductor (CMOS). In one example, sensor 2606 may represent a sensor with predetermined parameters, including but not limited to dynamic resolution range, linearity, and / or other characteristics specifically selected and / or designed for eye tracking.

[0227] As detailed above, the eye-tracking subsystem 2600 can generate one or more blinks. As detailed above, blink 2603 can represent a reflection of radiation from the user's eye structure (e.g., infrared radiation from an infrared source (e.g., source 2604)). In various embodiments, eye-tracking algorithms executed by a processor (located within or outside the AIVR) can be used to track blink 2603 and / or the user's pupils. For example, the AIVR may include a processor and / or storage devices to perform eye tracking locally, and / or include transceivers to send and receive data required for performing eye tracking on external devices (e.g., mobile phones, cloud servers, or other computing devices).

[0228] Figure 26 An example image 2605 acquired by an eye-tracking subsystem (e.g., eye-tracking subsystem 2600) is shown. In this example, image 2605 may include both the user's pupil 2608 and a flicker 2610 near the user's pupil. In some examples, artificial intelligence-based algorithms (e.g., computer vision-based algorithms) may be used to identify the pupil 2608 and / or the flicker 2610. In one embodiment, image 2605 may represent a single frame in a series of frames that can be continuously analyzed to track the user's eyes 2602. Furthermore, the pupil 2608 and / or the flicker 2610 may be tracked over a period of time to determine the user's gaze.

[0229] In one example, the eye-tracking subsystem 2600 can be configured to identify and measure the user's inter-pupillary distance (IPD). In some embodiments, the eye-tracking subsystem 2600 can measure and / or calculate the user's IPD while the user is wearing the artificial reality system. In these embodiments, the eye-tracking subsystem 2600 can detect the position of the user's eyes and can use this information to calculate the user's IPD.

[0230] As mentioned, the eye-tracking system or subsystem disclosed herein can track a user's eye position and / or eye movements in various ways. In one example, one or more light sources and / or optical sensors can acquire images of the user's eyes. The eye-tracking subsystem can then use the acquired information to determine the interpupillary distance, interocular distance, and / or 3D position (e.g., for distortion correction purposes) of each eye, including the magnitude of torsion and rotation (i.e., roll, pitch, and yaw) of each eye and / or the gaze direction of each eye. In one example, infrared light can be emitted by the eye-tracking subsystem and reflected from each eye. The reflected light can be received or detected by optical sensors and analyzed to extract eye rotation data from changes in the infrared light reflected by each eye.

[0231] An eye-tracking subsystem can use any of a variety of different methods to track a user's eyes. For example, a light source (e.g., an infrared LED) can emit a dot pattern onto each of the user's eyes. The eye-tracking subsystem can then detect and analyze the reflections of the dot pattern from each of the user's eyes (e.g., via optical sensors coupled to the artificial reality system) to identify the position of each of the user's pupils. Thus, the eye-tracking subsystem can track up to six degrees of freedom for each eye (i.e., 3D position, roll, pitch, and yaw) and can combine at least a subset of the tracking amounts from the user's eyes to estimate the gaze point (i.e., the 3D position or location in the virtual scene the user is looking at) and / or IPD.

[0232] In some cases, the distance between a user's pupil and the display can change when the user's eyes move to view in different directions. This change in distance between the pupil and the display when the viewing direction changes is known as "pupil migration," and because light focuses at different locations when the pupil-display distance changes, it can cause perceived distortion for the user. Therefore, measuring the distortion at different eye positions and pupillary distances relative to the display and generating distortion corrections for these different positions and distances allows for mitigation of distortion caused by pupil migration by tracking the 3D position of the user's eyes and applying distortion corrections corresponding to the 3D position of each eye at a given point in time. Thus, knowing the 3D position of each eye in the user's eyes allows for mitigation of distortion caused by changes in the distance between the pupil and the display by applying distortion corrections to each 3D eye position. Furthermore, as mentioned above, knowing the position of each eye in the user's eyes also enables the eye-tracking subsystem to automatically adjust the user's IPD (Interpupillary Distance).

[0233] In some embodiments, the display subsystem may include various additional subsystems that can operate in conjunction with the eye-tracking subsystem described herein. For example, the display subsystem may include a zoom subsystem, a scene rendering module, and / or a vergence-processing module. The zoom subsystem can cause the left and right display elements to change the focal length of the display device. In one embodiment, the zoom subsystem can physically change the distance between the display and the optics, through which the display can be viewed, by moving the display, the optics, or both. Alternatively, moving or translating two lenses relative to each other can also be used to change the focal length of the display. Therefore, the zoom subsystem may include actuators or motors that move the display and / or the optics to change the distance between them. The zoom subsystem may be separate from or integrated into the display subsystem. The zoom subsystem may also be integrated into or separate from the actuation subsystem and / or eye-tracking subsystem described herein.

[0234] In one example, the display subsystem may include a convergence processing module configured to determine the convergence depth of a user's gaze based on the gaze point and / or the estimated intersection of the gaze lines determined by the eye-tracking subsystem. Convergence can refer to the simultaneous movement or rotation of both eyes in opposite directions to maintain a single binocular vision, which can be performed naturally and automatically by the human eye. Therefore, the location where the user's eyes approach is the location the user is looking at, and is often also the location where the user's eyes are focused. For example, the convergence processing module may perform triangulation on the gaze line to estimate the distance or depth from the user, associated with the intersection of the gaze lines. The depth associated with the intersection of the gaze lines can then be used as an approximation of the adjustment distance, which can identify the distance between the location where the user's eyes are pointing and the user. Thus, the convergence distance can allow determination of the location where the user's eyes should focus, and the depth of the user's eyes from the point of focus, thereby providing information for rendering adjustments to the virtual scene (e.g., objects or focal planes).

[0235] The convergence processing module can collaborate with the eye-tracking subsystem described herein to adjust the display subsystem to take into account the user's convergence depth. When a user focuses on a distant object, their pupils may be slightly further apart than when focusing on a near object. The eye-tracking subsystem can acquire information about the user's convergence or depth of focus and can adjust the display subsystem to be closer when the user's eyes focus on or approach a near object, and to be further away when the user's eyes focus on or approach a distant object.

[0236] For example, the eye-tracking information generated by the aforementioned eye-tracking subsystem can also be used to modify various aspects of how different computer-generated images are presented. For instance, the display subsystem can be configured to modify at least one aspect of how computer-generated images are presented based on the information generated by the eye-tracking subsystem. For example, the computer-generated image can be modified based on the user's eye movements so that if the user looks up, the computer-generated image can move upwards on the screen. Similarly, if the user looks to the side or down, the computer-generated image can move to the side or down on the screen. If the user's eyes are closed, the computer-generated image can be paused or removed from the display and resumed once the user's eyes are open again.

[0237] The aforementioned eye-tracking subsystem can be incorporated in various ways into one or more of the various artificial reality systems described herein. For example, one or more components of the various components of system 2500 and / or eye-tracking subsystem 2600 can be incorporated into any of the augmented reality and / or virtual reality systems described herein to enable these systems to perform various eye-tracking tasks (including one or more of the multiple eye-tracking operations described herein).

[0238] As detailed above, the computing devices and systems described and / or illustrated herein broadly refer to any type or form of computing device or system capable of executing computer-readable instructions (e.g., those contained in the modules described herein). In the most basic configuration of these one or more computing devices, each of these one or more computing devices may include at least one storage device and at least one physical processor.

[0239] In some examples, the term "storage device" broadly refers to any type or form of volatile or non-volatile storage device or storage medium capable of storing data and / or computer-readable instructions. In one example, a storage device may store, load, and / or maintain one or more of the modules described herein. Examples of storage devices include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), optical disk drive, cache memory, variations or combinations of one or more of the above, or any other suitable storage memory.

[0240] In some examples, the term "physical processor" broadly refers to any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, a physical processor can access and / or modify one or more modules stored in the aforementioned storage device. Examples of physical processors include, but are not limited to, microprocessors, microcontrollers, central processing units (CPUs), field-programmable gate arrays (FPGAs) implementing soft-core processors, application-specific integrated circuits (ASICs), some of the above, variations or combinations of the above, or any other suitable physical processor.

[0241] Although shown as separate elements, the multiple modules described and / or illustrated herein may represent portions of a single module or application. Additionally, in some embodiments, one or more of these modules may represent one or more software applications or programs that, when executed by a computing device, enable the computing device to perform one or more tasks. For example, one or more of the multiple modules described and / or illustrated herein may represent a module stored on and configured to run on one or more of the multiple computing devices or systems described and / or illustrated herein. One or more of these modules may also represent all or part of one or more dedicated computers configured to perform one or more tasks.

[0242] Additionally, one or more of the modules described herein can convert data, physical devices, and / or representations of physical devices from one form to another. Alternatively or additionally, one or more of the modules described herein can convert the processor, volatile memory, non-volatile memory, and / or any other parts of a computing device from one form to another by executing on, storing data on, and / or otherwise interacting with the computing device.

[0243] In some embodiments, the term "computer-readable medium" broadly refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transport media and non-transitory media, such as carrier waves, magnetic storage media (e.g., hard disk drives, magnetic tape drives, and floppy disks), optical storage media (e.g., compact disks (CDs), digital video disks (DVDs), and Blu-ray discs), electronic storage media (e.g., solid-state drives and flash memory media), and other distribution systems.

[0244] The process parameters and order of steps described and / or illustrated herein are given by way of example only and may be changed as needed. For example, while multiple steps shown and / or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the shown or discussed order. The various exemplary methods described and / or illustrated herein may also omit one or more of the multiple steps described or illustrated herein, or may include additional steps in addition to those disclosed.

[0245] The foregoing description has been provided to enable others skilled in the art to optimally utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of this disclosure. The embodiments disclosed herein should be considered illustrative in all respects and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of this disclosure.

[0246] Unless otherwise stated, the terms “connected to” and “coupled to” (and their derivatives) as used in this specification and claims shall be construed as allowing both direct and indirect connections (i.e., via other elements or components). Additionally, the terms “a” or “an” as used in this specification and claims shall be construed as meaning “at least one of…”. Finally, for ease of use, the terms “comprising” and “having” (and their derivatives) as used in this specification and claims are interchangeable with the word “including” and have the same meaning as the word “comprising”.

Claims

1. A system comprising: a plurality of addressable display elements arranged in a plurality of display element rows and a plurality of display element columns; a plurality of clock signal lines; a plurality of gate drive units each coupled to one or more of the plurality of clock signal lines and configured to send gate signals to one or more of the plurality of display element rows based on clock signals provided on the one or more of the plurality of clock signal lines; and a controller configured to generate the clock signals to cause the plurality of gate drive units to send gate signals to respective ones of the plurality of display element rows, wherein the clock signals are controlled to adjust a number of gate signals sent simultaneously to the respective ones of the plurality of display element rows according to a foveated display pattern. the controller is configured to:

2. The system of claim 1, wherein, generate the clock signals during a first display period to cause a first one of the plurality of gate drive units to send gate signals to one or more of the plurality of display element rows during the first display period; and send respective display signals to each of the plurality of display element columns during the first display period. the controller is configured to, when the foveated display pattern includes a macro-pixel: generate the clock signals during the first display period to cause the first one of the plurality of gate drive units to send gate signals to at least two of the plurality of display element rows simultaneously during the first display period; and 3. The system of claim 2, wherein, send respective display signals to at least two of the plurality of display element columns simultaneously during the first display period. the respective display signals sent to the at least two of the plurality of display element columns simultaneously during the first display period are the same display signal.

5. The system of claim 4, wherein: the first one of the plurality of gate drive units sends gate signals to four of the plurality of display element rows simultaneously during the first display period; and 4. The system of claim 3, wherein, the controller is further configured to send the same display signal to four of the plurality of display element columns simultaneously during the first display period. the plurality of gate drive units includes a first gate driver having a plurality of clock inputs coupled to a first subset of the plurality of clock signal lines and a second gate driver having a plurality of clock inputs coupled to a second subset of the plurality of clock signal lines, the second subset being different than the first subset. at least one of the plurality of clock inputs of a respective one of the plurality of gate drivers enables or disables functionality of the respective gate driver with respect to sending one or more gate signals. the controller, the plurality of gate drive units, and the plurality of clock signal lines are implemented as an integrated circuit.

6. The system of claim 1, wherein, 9. A method comprising:

7. The system of claim 6, wherein, ​ 8. The system of claim 1, wherein, ​ ​ sending a gate signal to each of a first number of rows of the plurality of display elements simultaneously by a first gate drive unit, the first gate drive unit receiving a first clock signal from a first subset of the plurality of clock signal lines; sending a gate signal to each of a second number of rows of the plurality of display elements simultaneously by a second gate drive unit, the second gate drive unit receiving a second clock signal from a second subset of the plurality of clock signal lines, the second subset being different from the first subset, wherein the second number of rows is different from the first number of rows.

10. The method of claim 9, further comprising: adjusting a phase of the first clock signal to produce the second clock signal.

11. The method of claim 9, further comprising: generating the first clock signal and the second clock signal from image data by a controller.

12. The method of claim 9, wherein, the plurality of display elements are arranged into a plurality of display element columns and a plurality of display element rows; and wherein the method further comprises: directing the gate signal to each of the first number of rows of the plurality of display elements simultaneously with directing a display signal to one or more of the plurality of display element columns, thereby simultaneously directing both the display signal and the gate signal to one or more display elements.

13. A system comprising: a plurality of addressable display elements arranged into a plurality of display element rows and a plurality of display element columns; and a controller comprising a plurality of de-multiplexers and configured to, in dependence on image data received by the controller: relay first display data to a first number of the plurality of display element columns simultaneously by a first de-multiplexer of the plurality of de-multiplexers; and relay second display data to a second number of the plurality of display element columns simultaneously by a second de-multiplexer of the plurality of de-multiplexers after relaying the first display data to the first number of the plurality of display element columns, the second number of the plurality of display element columns being different from the first number of the plurality of display element columns.

14. The system of claim 13, wherein, the image data comprises a plurality of de-multiplexing indicators; and wherein the controller is configured to select one of the plurality of de-multiplexers to receive display data based on one of the plurality of de-multiplexing indicators.

15. The system of claim 14, wherein, the controller is configured to, based on the image data comprising a first de-multiplexing indicator, first image data, a second de-multiplexing indicator, and second image data, send the first display data to the first de-multiplexer based on the first de-multiplexing indicator and the first image data, and send the second display data to the second de-multiplexer based on the second de-multiplexing indicator and the second image data.

16. The system of claim 15, wherein, the first number of the plurality of display element columns is greater than one in number; and wherein the second number of the plurality of display element columns is one in number.

17. The system of claim 13, wherein, the plurality of display element columns are configured with alternating polarity; and wherein the first de-multiplexer comprises: a first input configured to route a first portion of the first display data to a first column of display elements of the plurality of columns of display elements; and a second input configured to route a second portion of the first display data to a second column of display elements of the plurality of columns of display elements, the second column of display elements being adjacent to the first column of display elements and having an opposite polarity to the first column of display elements.

18. The system of claim 17, wherein, the controller is configured to generate the first portion of the first display data, the first portion of the first display data being a polarity-inverted version of the second portion of the first display data.

19. A method comprising: relaying, by a first de-multiplexer of a plurality of de-multiplexers, first display data to a first number of columns of display elements of a plurality of columns of display elements of a display simultaneously; and after relaying the first display data to the first number of columns of display elements of the plurality of columns of display elements, relaying, by a second de-multiplexer of the plurality of de-multiplexers, second display data to a second number of columns of display elements of the plurality of columns of display elements simultaneously, the second number of columns of display elements of the plurality of columns of display elements being different from the first number of columns of display elements of the plurality of columns of display elements. selecting, by a controller, one of a plurality of de-multiplexers including the first de-multiplexer and the second de-multiplexer to receive display data based on one of a plurality of de-multiplexing indicators from image data including the plurality of de-multiplexing indicators.

20. The method of claim 19, further comprising: the first number of columns of display elements of the plurality of columns of display elements is greater than one; and wherein the second number of columns of display elements of the plurality of columns of display elements is one.

21. The method of claim 19, wherein, relaying the first display data to the first number of columns of display elements of the plurality of columns of display elements includes relaying the first display data to a plurality of columns of display elements of the plurality of columns of display elements configured to produce light of a same color.

22. The method of claim 19, wherein, ​

Images