Camera-based facial feature tracking with distributed compute

Abstract

The present disclosure is directed to augmented and / or virtual reality (AR / VR) devices having a "distributed compute" architecture using an on-sensor compute architecture and / or a near-sensor compute architecture in the facial feature tracking system of AR / VR devices, and / or any outward-facing sensor system. Processing data at or near the source (the sensor) may drastically reduce data transfer times and alleviate bandwidth constraints. This localized processing can ensure that more pertinent information is relayed to the central system(s), optimizing both speed and efficiency. Furthermore, such distributed compute architectures may provide enhanced power efficiency, a highly desirable characteristic for a wearable or portable AR / VR device, which may thereby ensure longer than typical operational times without compromising performance.

Description

CAMERA-BASED FACIAL FEATURE TRACKING WITH DISTRIBUTED COMPUTECROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the filing benefit of U.S. Provisional Application No. 63 / 649,139 filed 17 May 2024.TECHNICAL FIELD

[0002] This patent application relates generally to using distributed compute architectures and / or techniques for eye / face tracking and / or outward-facing sensing in a near-eye device, and in particular to implementing on-sensor compute architecture and / or near-sensor compute architecture architectures for inward-facing and / or outward-facing sensing in a near-eye augmented reality (AR) / virtual reality (VR) display device.BACKGROUND

[0003] With recent advances in technology, the prevalence and proliferation of content creation and delivery has increased greatly in recent years. In particular, interactive content such as augmented reality (AR) content (where the digital content may be combined in real-time with the user's vision of the user's real world ambient environment, sometimes referred to as "Mixed Reality (MR)"), virtual reality (VR) content, and / or any other content within and associated with a real and / or virtual environment (e.g., a "metaverse") has become appealing to consumers. To facilitate delivery of this and other related content, service providers have endeavored to provide various forms of wearable display systems.

[0004] One such example may be a head-mounted display (HMD) device, such as a wearable eyewear, a wearable headset, or eyeglasses. In some examples, the head-mounted display (HMD) device may project or direct light to may display virtual objects or combine images of real objects with virtual objects, as in augmented reality (AR) and / or virtual reality (VR) and / or applications. For example, in an augmented reality (AR) system, a user may view both images of virtual objects (e.g., computer-generated images (CGIs)) and the surrounding environment. Head-mounted display (HMD) devices may also present interactive content, where a user's (wearer's) gaze may be used as input for the interactive content.

[0005] Eye and / or face tracking may be especially important in augmented reality / virtual reality (AR / VR) environments, because real-time processing in determining the user's gaze direction, eye movement, facial expression, etc., may be highly desirable to ensure accurate and seamless user interactions with the AR / VR environment and real-time and seamless AR / VR display to the user, as well as providing an optimal user experience (UX)generally. Traditional near-eye AR / VR display devices may introduce latency and bottlenecks due to the sheer volume of raw data from the facial feature tracking sensors which may be processed as quickly as possible to provide an optimal user experience (UX). Accordingly, faster and more accurate data processing in facial feature tracking systems in near-eye AR / VR display devices is a highly desirable characteristic.

[0006] The present disclosure seeks to address, at least in part, any or all of the drawbacks and disadvantages described above.SUMMARY

[0007] According to a first aspect of the present disclosure there is provided a system, comprising: an integrated circuit in a near-eye device; a facial feature tracking sensor configured to sense a user's eye or surrounding facial features from the near-eye device; and an on-sensor compute architecture configured to receive sensor raw data from the facial feature tracking sensor, pre-process the sensor raw data, and transmit pre-processed facial feature tracking sensor data to a facial feature tracking main processing unit.

[0008] In some embodiments, the facial feature tracking sensor may be directed to at least one of a user's eye, facial features surrounding the user's eye, or a waveguide configured to propagate light to and from the facial feature tracking sensor and a facial feature being tracked.

[0009] In some embodiments, the on-sensor compute architecture may be configured to optimize efficient power conversion and gaze vectors storage.

[0010] In some embodiments, the system may further comprise a near-sensor compute architecture disposed separately from the on-sensor compute architecture.

[0011] In some embodiments, the on-sensor compute architecture and the nearsensor compute architecture may comprise a distributed compute architecture.

[0012] In some embodiments, the distributed compute architecture may comprise employing the on-sensor compute architecture, the near-sensor compute architecture, or both the on-sensor compute architecture and the near-sensor compute architecture as at least a portion of a contextual artificial intelligence (CAI) system.

[0013] In some embodiments, the system may further comprise an outward-facing sensor configured to sense an environment external to the system.

[0014] In some embodiments, the outward-facing sensor may be configured to perform within the distributed compute architecture.

[0015] In some embodiments, the near-eye device may comprise a frame, wherein the facial feature tracking sensor may be disposed in the frame and the facial feature tracking main processing unit may be disposed in the frame.

[0016] In some embodiments, the near-sensor compute architecture may be disposed in the frame.

[0017] In some embodiments, the on-sensor architecture may comprise disposing the integrated circuit or a module together with the facial feature tracking sensor on a chip.

[0018] According to a second aspect of the present disclsoure there is provided a system, comprising: a near-eye augmented reality display device, comprising: a frame; a facial feature tracking sensor disposed in the frame configured to sense a user's facial feature and produce facial feature tracking sensor raw data; an integrated circuit disposed separately in the frame and operatively connected to the facial feature tracking sensor, comprising: an outward-facing sensor configured to sense an external environment and produce outwardfacing sensor raw data; and an on-sensor compute architecture disposed on the facial feature tracking sensor configured to: receive the outward-facing sensor raw data; receive the facial feature tracking sensor raw data; pre-process both the outward-facing sensor raw data and the facial feature tracking sensor raw data; and transmit the pre-processed outward-facing and facial feature tracking sensor raw data; and an augmented reality main processing unit configured to receive the pre-processed outward-facing and facial feature tracking sensor raw data and to perform augmented reality processing.

[0019] In some embodiments, the facial feature tracking sensor may be directed to at least one of a user's eye, facial features surrounding the user's eye, or a waveguide configured to propagate light to and from the facial feature tracking sensor and a facial feature being tracked.

[0020] In some embodiments, the on-sensor compute architecture may be configured to optimize efficient power conversion and gaze vectors storage.

[0021] In some embodiments, the system may further comprise a plurality of facial feature tracking sensors disposed in the frame configured to sense a user's eye or surrounding facial tissue and produce raw facial feature tracking sensor raw data.

[0022] In some embodiments, the system may further comprise a plurality of outward-facing sensors disposed in the frame to sense the external environment and to produce raw outward-facing sensor raw data.

[0023] In some embodiments, the system may further comprise a near-sensor compute architecture disposed separately within the frame and operatively connected to the facial feature tracking sensor and the outward-facing sensor, wherein the near-sensor compute architecture may be configured to: receive the outward-facing sensor raw data from the outward-facing sensor; receive the facial feature tracking sensor raw data from the facial feature tracking sensor; pre-process both the outward-facing sensor raw data and the facial feature tracking sensor raw data; and transmit the pre-processed outward-facing sensor raw data and facial feature tracking sensor raw data to the augmented reality main processing unit.

[0024] According to a third aspect of the present disclosure there is provided a method, comprising: sensing facial feature raw data from at least one facial feature tracking sensor disposed on a chip in a frame of a near-eye display device; receiving the facial feature tracking sensor raw data from the at least one facial feature tracking sensor; pre-processing the facial feature tracking sensor raw data using an integrated circuit co-disposed on the chip to provide pre- processed facial feature tracking sensor raw data; and transmitting the pre- processed facial feature tracking sensor raw data to a main processing unit operatively coupled to the chip.

[0025] In some embodiments, the method may further comprise receiving outwardfacing sensor raw data from at least one outward-facing sensor disposed on the frame of the near-eye display device.

[0026] In some embodiments, the method may further comprise receiving pre- processed data from a near-sensor compute architecture disposed in the frame of the near- eye display device.

[0027] It will be appreciated that any features described herein as being suitable for incorporation into one or more aspects or embodiments of the present disclosure are intended to be generalizable across any and all aspects and embodiments of the present disclosure. Other aspects of the present disclosure can be understood by those skilled in the art in light of the description, the claims, and the drawings of the present disclosure. The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims.BRIEF DESCRIPTION OF DRAWINGS

[0028] Features of the present disclosure are illustrated by way of example and notlimited in the following figures, in which like numerals indicate like elements. One skilled in the art will readily recognize from the following that alternative examples of the structures and methods illustrated in the figures can be employed without departing from the principles described herein.

[0029] FIG. 1 illustrates a block diagram of a near-eye display device which may form part of a display system environment, according to one or more embodiments of the present disclosure.

[0030] FIGS. 2A and 2B illustrate a front prospective view and a back prospective view, respectively, of a near-eye display device in the form of a head-mounted display (HMD) device, according to one or more embodiments of the present disclosure.

[0031] FIGS. 3A and 3B illustrate a perspective view and a top view, respectively, of a near-eye display device in the form of a pair of glasses, according to one or more embodiments of the present disclosure.

[0032] FIG. 4A is a block diagram of a facial feature tracking sensor with an on-sensor compute, where the facial feature tracking sensor is pointed towards the user's face, according to one or more embodiments of the present disclosure.

[0033] FIG. 4B is a block diagram of a facial feature tracking sensor with an on-sensor compute, where the facial feature tracking sensor is pointed into the display waveguide of the near-eye display device, according to one or more embodiments of the present disclosure.

[0034] FIG. 5 is a block diagram of a facial feature tracking sensor operably connected to a near-sensor compute, which, in turn, is connected to the facial feature tracking main processing system, according to one or more embodiments of the present disclosure.

[0035] FIG. 6 is a block diagram of an integrated module including an outward-facing sensor with an on-sensor compute, where the integrated outward-facing sensor with an on- sensor compute architecture module is operatively connected to a facial feature tracking sensor pointed towards the user's face, according to one or more embodiments of the present disclosure.

[0036] FIG. 7 is a block diagram of a monolithic integrated circuit including both an on-sensor compute architecture and dual inward / outward-facing sensors, where the integrated inward-facing sensor (which may be, e.g., a facial feature tracking sensor) is pointed towards the user's face, according to one or more embodiments of the present disclosure.

[0037] FIG. 8 is a block diagram of a monolithic integrated circuit including both an on-sensor compute architecture and dual inward / outward-facing sensors, where the integrated inward-facing sensor (which may be, e.g., a facial feature tracking sensor) is pointed into the display waveguide of the near-eye display device, according to one or more embodiments of the present disclosure.

[0038] FIG. 9 is a block diagram of an integrated module including an outward-facing sensor with an on-sensor compute, where the integrated module is operatively connected to a network of facial feature tracking sensors pointed towards the user's face, according to one or more embodiments of the present disclosure.

[0039] FIG. 10 is a block diagram of a distributed compute architecture where a nearsensor compute architecture module is operatively connected to both a network of facial feature tracking sensors pointed towards the user's face and a network of outward-facing sensors pointed towards the external environment, according to one or more embodiments of the present disclosure.

[0040] FIG. 11 is a flowchart illustrating a method for an on-sensor compute architecture and / or near-sensor compute architecture to combine and / or otherwise pre- process sensor raw data from both one or more inward-facing sensors and one or more outward-facing sensors, according to one or more embodiments of the present disclosure.

[0041] Throughout the drawings, identical reference characters and descriptions indicate similar, but not necessarily identical, elements. While the exemplary embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. However, the exemplary embodiments described herein are not intended to be limited to the particular forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims.DETAILED DESCRIPTION

[0042] For simplicity and illustrative purposes, the present application is described by referring mainly to examples thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. It will be readily apparent, however, that the present application may be practiced without limitation to these specific details. In other instances, some methods and structures readily understood by one of ordinary skill in the art have not been described in detail so as not tounnecessarily obscure the present application. As used herein, the terms "a" and "an" are intended to denote at least one of a particular element, the term "includes" means includes but not limited to, the term "including" means including but not limited to, and the term "based on" means based at least in part on.

[0043] As used herein, a "wearable device" may refer to any portable electronic device that may be worn on any body part of a user and used to present audio and / or video content, control other devices, monitor bodily functions, and / or perform similar actions. As used herein, a "near-eye device" may refer to a device that may be in close proximity to a user's eye and may have optical capabilities, whereas a "near-eye display device" may refer to a device that may be in close proximity to a user's eye and may be capable of some sort of display to one or both of the user's eyes. Accordingly, a near-eye display device may be a head-mounted display (HMD) device, such as a wearable eyewear, a wearable headset, and / or "smartglasses," which may be used for interacting with virtual reality (VR), augmented reality (AR), and / or any environment of real and / or virtual elements, such as a "metaverse." As used herein, a "near-eye AR / VR display device" may refer to a near-eye display device which may be used to display and / or for interact with any virtual reality (VR) and / or augmented reality (AR) content, including, but not limited to, any virtual reality (VR) and / or augmented reality (AR) environment (such as a metaverse). As used herein, a "user" may refer to a user or wearer of a "wearable device," "near-eye device," "near-eye display device," and / or "near-eye AR / VR display device," depending on the context, which would be clear to one of ordinary skill in the art.

[0044] As mentioned above, the speed at which a near-eye AR / VR display device may process and utilize data from its sensors to present interactive AR / VR content (and / or an AR / VR environment) to its user may be important to overall effectiveness, efficiency, and the user experience (UX). More particularly, the speed at which the data from facial feature tracking sensors may be processed and utilized may be key to the effective, accurate, and seamless presentation of AR / VR content (and / or an AR / VR environment) to the user of a near-eye AR / VR display device.

[0045] However, near-eye AR / VR display devices may typically have data latency and processing bottlenecks in getting the raw data (e.g., facial feature tracking sensor raw data and / or outward-facing sensor raw data) from the facial feature tracking sensors to the AR / VR system architecture, which may prevent accurate and seamless user interactions with theAR / VR content / environment, as well as generally degrading the user experience (UX). Typical architectures, where raw data from sensors is transmitted to a main processor for analysis, introduce latency due to data transfer and processing bottlenecks. Additionally, the sheer volume of high-resolution data generated by the sensors may strain the bandwidth and overall capabilities of the internal data transfer system, leading to inefficiencies and potential data loss, which may compromise other systems as well as the accuracy and responsiveness of the AR / VR content / environment system.

[0046] According to examples of the present disclosure, a "distributed compute" approach is described for, e.g., facial feature tracking systems in near-eye AR / VR display devices, where some of the processing typically performed at a main processing unit may be moved to other processing components nearer to the facial feature tracking sensors. In some examples, an "on-sensor" architecture may be employed, where some of the sensor raw data is processed on the same chip, integrated circuit, module, and / or component as the sensor itself. In some examples, a "near-sensor" architecture may be employed, where some of the sensor raw data is processed on a chip, integrated circuit, module, and / or component near to the sensor within the frame of the near-eye AR / VR display device. In such examples, it may be possible that any chip, integrated circuit, module, and / or component within the frame of the near-eye display device may be considered "near-sensor" if, for example, some data, image, and / or AR / VR environment processing is processed outside of the near-eye display device.

[0047] In systems, apparatus, and / or methods according to the present disclosure, the "distributed compute" architecture using an on-sensor compute architecture and / or a near-sensor compute architecture in the facial feature tracking system and / or any outwardfacing sensor system. By processing data at or near the source (the sensor), these on- sensor / near-sensor architectures may drastically reduce data transfer times and alleviate bandwidth constraints. This localized processing ensures that more pertinent information is relayed to the central system(s), optimizing both speed and efficiency. Furthermore, such distributed compute architectures may provide enhanced power efficiency, a highly desirable characteristic for a wearable or portable AR / VR device, which may thereby ensure longer than typical operational times without compromising performance.

[0048] In some examples of the present disclosure, a "distributed compute" architecture using an on-sensor compute architecture and / or a near-sensor computearchitecture may be employed as part of a Contextual Artificial Intelligence (CAI) system implemented in a near-eye device, where one or more on-sensor compute architectures and / or near-sensor compute architecture disposed separately from the on-sensor compute architecture assist in combining / pre-processing sensor raw data from a multitude of different sensors in order to infer more complex datapoints. In some examples, such a system may be continually gathering data all day, and the complex datapoints may include the user's activities and / or interactions with people and the external environment over the course of the day.

[0049] While some advantages and benefits of the present disclosure are discussed herein, there are additional benefits and advantages which would be apparent to one of ordinary skill in the art.

[0050] The following disclosure is broken down into 2 main sections:I. Near-Eye Display Device(s), describing near-eye display devices which may be employed with examples of the present disclosure, with reference to FIGS. 1-3B; andII. Facial feature Tracking Sensor System with Distributed Compute: On-Sensor and / or Near-Sensor, describing non-limiting examples of facial feature tracking sensors with an on-sensor compute, with reference to FIGS. 4A-4B; a facial feature tracking sensor operably connected to a near-sensor compute, which, in turn, is connected to the facial feature tracking main processing system, with reference to FIG. 5; an integrated module including an outward-facing sensorwith an on-sensor compute, operatively connected to a facial feature tracking sensor, with reference to FIG. 6; a monolithic integrated circuit including both an on-sensor compute architecture and dual inward / outward-facing sensors, with reference to FIGS. 7-8; an integrated module including an outward-facing sensor with an on-sensor compute, operatively connected to a network of facial feature tracking sensors, with reference to FIG. 9; a near-sensor compute architecture operatively connected to both a network of facial feature tracking sensors and a network of outward-facing sensors, with reference to FIG. 10; and a flowchart illustrating a method for an on-sensor compute architecture and / or near-sensor compute architecture to combine and / or otherwise pre-process sensor raw data from inward-facing sensors and outward-facing sensors, with reference to FIG. 11.I. NEAR-EYE DISPLAY DEVICE(S)

[0051] FIG. 1 illustrates a block diagram of a near-eye display device which may be part of a display system environment, according to an example. As mentioned above, a "near- eye display device" may refer to a device in close proximity to a user's eye which is capable of some sort of display to one or both of the user's eyes, including a wearable headset, such as, e.g., a head-mounted display (HMD) device, and / or other wearable eyewear, such as, e.g., "smartglasses." The display environment may include an artificial reality environment, where "artificial reality" may refer to aspects of, among other things, a "metaverse" or an environment of real and virtual elements and may include use of technologies associated with virtual reality (VR) and / or augmented reality (AR), which is sometimes also referred to as mixed reality (MR). As used herein a "user" may refer to a user or wearer of a "near-eye display device."

[0052] While this section describes near-eye devices (and, in particular, near-eye display devices), examples of the present disclosure are not limited thereto. For instance, examples of the present disclosure may apply to near-eye devices without specific image displaying capabilities, such as, for example, the Ray-Ban™ | Meta™ line of smartglasses. Moreover, examples of the present disclosure are expressly intended to apply to other wearable devices (as defined above) besides the near-eye devices described herein, including other wearable computing platforms, which may have, e.g., Internet of Things (loT), audio / visual, health monitoring, WiFi and radio reception, and / or other capabilities, such as smartwatches, compute "pucks," as would be understood by one of ordinary skill in the art.

[0053] As shown in FIG. 1, an artificial reality system environment 100 may include a near-eye display device 120 and an optional input / output interface 140, each of which may be coupled to an optional console 110. The artificial reality system environment 100 may also include an optional external imaging device (not shown), as discussed in relation to locators 126 below. As would be understood by one of ordinary skill in the art, FIG. 1 is a schematic diagram, and is not indicative of size, location, orientation, and / or relative sizes / locations / orientations of any of the systems, components, and / or connections shown therein. For example, a figurative "bus" connects some, but not all, of the components shown inside the near-eye display device 120 in FIG. 1; however, all of the components therein may be connected by the same bus and / or busses, or may have direct and / or indirect connections with, e.g., the processor(s) 121. Such electrical, control, and / or power connections may beimplemented in a large variety of ways, as would be understood by one of ordinary skill in the art.

[0054] The optional console 110 may be optional in some instances where functions of the optional console 110 may be integrated into the near-eye display device 120. In some examples, the near-eye display device 120 may be implemented in any suitable form-factor, including a head-mounted display (HMD), a pair of glasses, or other similar wearable eyewear or device. In some examples, the near-eye display device 120 may include one or more rigid bodies, which may be rigidly or non-rigidly coupled to each other. In some examples, a rigid coupling between rigid bodies may cause the coupled rigid bodies to act as a single rigid entity, while in other examples, a non-rigid coupling between rigid bodies may allow the rigid bodies to move relative to each other. Some non-limiting specific examples of implementations of the near-eye display device 120 are described further below with respect to FIGS. 2A-2B and 3A-3B.

[0055] In some examples, the near-eye display device 120 may present content to a user, including, for example, audio / visual content, such as, e.g., virtual reality (VR), augmented reality (AR), sometimes referred to as mixed reality (MR) content. In augmented reality (AR) examples, the near-eye display device 120 may combine images (and / or a see- through view) of a physical, real-world environment external to the near-eye display device 120 and artificial reality / digital content (e.g., computer-generated images, video, sound, etc.) to present an augmented reality (AR) environment for the user.

[0056] As shown in FIG. 1, the near-eye display device 120 may include any one or more of one or more processor(s) 121, display electronics 122, one or more outward-facing sensor(s) 123, display optics 124, one or more locators 126, one or more position sensors 128, a facial feature tracking unit 130, an inertial measurement unit (IMU) 132, a wireless communication sub-system 134, one or more outward projectors 172, and / or one or more inward projectors 173. In some examples, the near-eye display device 120 may include additional components; in other examples, the near-eye display device 120 may omit any one or more of the one or more locators 126, the one or more position sensors 128, the facial feature tracking unit 130, the inertial measurement unit (IMU) 132, the wireless communication sub-system 134, the one or more outward projectors 172, and / or the one or more inward projectors 173. As would be understood by one of ordinary skill in the art, various operational, electronic, communication (for, e.g., control signals), electrical and othersuch connections may or may not also be included between and among the components of the near-eye display device 120.

[0057] In some examples, the display electronics 122 may display or facilitate the display of images to the user according to data received from control electronics disposed in, for example, the near-eye display device 120, the optional console 110, the input / output interface 140, and / or a system connected by wireless or wired connection with the near-eye display device 120. In some examples, such electronics may include a virtual reality engine, such as, for example, the virtual reality engine 116 in the external console 110 described below, a virtual reality engine implemented, in part or in whole, in electronics in the near-eye display device 120, and / or a virtual reality engine implemented, in whole or in part, in an external system connected by the wireless communication subsystem 134, etc. In some examples, the display electronics 122 may include one or more display panels, and may include and / or be operationally connected to the display optics 124. In some examples, the display electronics may include one or more of a liquid crystal display (LCD) and / or a lightemitting diode (LED) and may include any number of pixels to emit light of a predominant color such as red, green, blue, white, or yellow. In some examples, the display electronics 122 may display a three-dimensional (3D) image, e.g., using stereoscopic effects produced by two- dimensional panels, to create a subjective perception of image depth.

[0058] In some examples, the display electronics 122 may include and / or be operationally connected to the one or more outward projectors 172 and / or the one or more inward projectors 173; in some examples, the facial feature tracking unit 130 may also include and / or be operationally connected to the one or more inward projectors 173. As indicated by the striped lined box in FIG. 1, there may be operational and / or other connections between and among the display electronics 122, the facial feature tracking unit 130, the one or more outward projectors 172, and / or the one or more inward projectors 173. As indicated above, such connections may also be included between and among these and other components of the near-eye display device 120; the possible connections indicated by the striped lined box in FIG. 1 are shown herein as they are germane to examples of the present disclosure.

[0059] In some examples, the one or more outward-facing sensor(s) 123 may include, e.g., a camera, an image sensor, such as a complementary metal-oxide semiconductor (CMOS) image sensor, a defocused image sensor, a light field sensor, a single photon avalanche diode (SPAD), and / or, in certain implementations, a non-imaging sensor, such as aself-mixing interferometer (SMI) sensor. In some examples, the one or more outward-facing sensor(s) 123 may be a combined VCSEL / SMI integrated circuit which may be employed as both a light source and a sensor. In some examples, the one or more outward-facing sensor(s) 123 may be employed for purposes of creating a user-responsive AR / VR display environment by sensing the external environment in relation to the user, such as in, for example, the outward-facing camera 250 in the head-mounted display (HMD) device 200 in FIGS. 2A-2B and / orthe outward-facing camera(s) 320 in FIGS. 3A-3B as discussed and described more fully below.

[0060] In some examples, the one or more inward projectors 173 may, under the control of the display electronics 122, form an image in angular domain for direct observation by a viewer's eye through a pupil. In some examples, the same or different one or more inward projectors 173 may, under the control of the facial feature tracking unit 130, project a fringe or other pattern on the eye and / or other portions of the user's face (such as the inward projectors 310 of FIGS. 3A and 3B discussed below). As used herein, "facial feature tracking" may refer to determining an eye's position or relative position, including orientation, location, and / or gaze of a user's eye, as well as determining facial characteristics and parameters, such as from the flesh covering the orbital socket, the eyelids, eye brows, and / or any other regions around the eye or optionally elsewhere on the face. In examples where at least some of the one or more inward projectors 173 may be used to project a fringe pattern on the eye and / or face, reflections from the projected pattern on the eye may be captured by a camera and analyzed (e.g., by the facial feature tracking unit 130 and / or the facial feature tracking module 118 in the optional console 110) to determine a position of the eye (the pupil), a gaze, etc., and / or characteristics of one or more portions of the face (including the region immediately adjacent to the eye). In other examples, the facial feature tracking unit 130 may capture reflected radio waves emitted by a miniature radar unit. These data associated with the eye and / or face may be used to determine or predict eye position, orientation, movement, location, gaze, etc., and / or characteristics of one or more portions of the face (including the region immediately adjacent to the eye).

[0061] In some examples, the one or more outward projectors 172 may, under the control of the display electronics 122, project a fringe or other pattern on the external environment (such as the outward projectors 315 of FIGS. 3A and 3B). In examples where at least some of the one or more outward projectors 172 may be used to project a fringe patternon the external environment, reflections from the projected pattern on the external environment may be captured by a camera and analyzed to determine a position of objects in the external environment, distances between the user and objects and / or surfaces of the external environment, etc.

[0062] In some examples, a location of any of the one or more inward projectors 173 and / or the one or more outward projectors 172 may be adjusted to enable any number of design modifications. For example, in some instances, the one or more inward projectors 173 may be disposed in the near-eye display device 120 in front of the user's eye (e.g., "frontmounted" placement). In a front-mounted placement, in some examples, the one or more inward projectors 173 under control of the display electronics 122 may be located away from a user's eyes (e.g., "world-side"). In some examples, the near-eye display device 120 may utilize a front-mounted placement to propagate light and project an image on the user's eye(s).

[0063] In some examples, the one or more outward and / or inward projectors 172 and / or 173 may employ a controllable light source (e.g., a laser) and a microelectromechanical system (MEMS) beam scanner to create a light field from, for example, a collimated light beam. In some examples, the light source of the one or more projectors 172 and / or 173 may include one or more of a Vertical Cavity Surface Emitting Laser (VCSEL), liquid crystal display (LCD), a light emitting diode (LED) or micro-light emitting diode (mLED), an organic light emitting diode (OLED), an inorganic light emitting diode (ILED), an active-matrix organic light emitting diode (AMOLED), a transparent organic light emitting diode (TLED), any other suitable light source, and / or any combination thereof. In some examples, the one or more projectors may include a single electronic display or multiple electronic displays (e.g., one for each eye of the user).

[0064] In some examples, the display optics 124 may project, direct, and / or otherwise display image content optically and / or magnify image light received from the one or more inward projectors 173 (and / or otherwise created by the display electronics 122), correct optical errors associated with image light created and / or received from the external environment, and / or present the (corrected) image light to a user of the near-eye display device 120. In some examples, the display optics 124 may include an optical element or any number of combinations of various optical elements as well as mechanical couplings to, for example, maintain relative spacing and orientation of the optical elements in thecombination.

[0065] In some examples, the display optics 124 may include one or more of a beamforming element, a beam-shaping element, an aperture, a Fresnel lens, a refractive element (such as, e.g., a lens), a reflective element (such as, e.g. a mirror), a diffractive element, a polarization element, a waveguide, a filter, or any other optical element suitable for affecting and / or otherwise manipulating light emitted from the one or more inward projectors 173 (and / or otherwise created by the display electronics 122). In some examples, the display optics 124 may include an optical coating, such as an anti-reflective coating, a reflective coating, a filtering coating, and / or a combination of different optical coatings. In some examples, the display optics 124 may include a Pancharatnam-Berry phase (PBP) or other phase-modification elements, a surface grating, a high-contrast grating, diffractive gratings (such as, e.g. Polarization Volumetric Hologram-based (PVH) gratings, Surface Relief Gratings (SRGs), Volume Bragg Gratings (VBGs), a diffractive optical element (DOE), etc.), nano-optics (including, e.g., metalenses and metasurfaces), micro-structures (including those fabricated using 3D printing), a liquid lens, a mask (such as, e.g., a phase mask), surface coatings, lithographically-created layered waveguides, and / or any other suitable technology, layer, coating, and / or material feasible and / or possible either presently or in the future, as would be understood by one of ordinary skill in the art.

[0066] For additional details concerning the architecture and constructions of metasurfaces, metalenses, and nonlocal flat optics which may be employed in the optics, projectors, sensors, and other components of examples of the present disclosure, see, for example, Zheng et al., Compound Meta-Optics for Complete and Loss-Less Field Control, ACS Nano 2022, 16, 15100-15107; https: / / doi.org / 10.1021 / acsnano.2c06248; which discusses multilayer optical metasurfaces in the design space of flat optics which offer compact platforms for the manipulation of the amplitude, phase, and / or polarization state of light; Shastri & Monticone, Nonlocal Flat Optics, Nature Photonics, vol. 17, pp. 36-47 (22 Dec. 2022); https: / / doi.org / 10.1038 / s41566-022-01098-5; and Chen, A. & Monticone, F., Dielectric Nonlocal Metasurfaces for Fully Solid-State Ultrathin Optical Systems, ACS Photonics, vol. 8, issue 5, pp. 1439-1447 (2021).

[0067] In some examples, the display optics 124 may be used to combine the view of an environment external to the near-eye display device 120 and artificial reality content (e.g., computer-generated images) generated by, e.g., the virtual reality engine 116 in the console110, and projected by, e.g., the one or more inward projectors 173 (and / or otherwise created by the display electronics 122). In such examples, the display optics 124 may augment images of a physical, real-world environment external to the near-eye display device 120 with generated and / or overlaid digital content (e.g., images, video, sound, etc.) projected by the one or more inward projectors 173 (and / or otherwise created by the display electronics 122) to present augmented reality (AR) content to a user.

[0068] In some examples, the display optics 124 may also be designed to correct one or more types of optical errors, such as two-dimensional optical errors, three-dimensional optical errors, or any combination thereof. Examples of two-dimensional errors may include barrel distortion, pincushion distortion, longitudinal chromatic aberration, and / or transverse chromatic aberration. Examples of three-dimensional errors may include spherical aberration, chromatic aberration field curvature, and astigmatism.

[0069] In some examples, the one or more locators 126 may be objects located in specific positions relative to one another and relative to a reference point on the near-eye display device 120. In some examples, the optional console 110 may identify the one or more locators 126 in images captured by an optional external imaging device to determine the artificial reality headset's position, orientation, or both. The one or more locators 126 may each be a light-emitting diode (LED), a corner cube reflector, a reflective marker, a type of light source that contrasts with an environment in which the near-eye display device 120 operates, or any combination thereof.

[0070] In some examples, the optional external imaging device (not shown) may include one or more cameras, one or more video cameras, any other device capable of capturing images including the one or more locators 126, or any combination thereof. The optional external imaging device may detect light emitted or reflected from the one or more locators 126 in a field of view of the optional external imaging device.

[0071] In some examples, the one or more position sensors 128 may sense motion of the near-eye display device 120 and, in response, generate one or more measurement signals and / or data. Examples of the one or more position sensors 128 may include any number of accelerometers, gyroscopes, magnetometers, and / or other motion-detecting or errorcorrecting sensors, or any combination thereof.

[0072] In some examples, the inertial measurement unit (IMU) 132 may be an electronic device that generates fast calibration data based on measurement signals receivedfrom the one or more position sensors 128. The one or more position sensors 128 may be located external to the inertial measurement unit (IMU) 132, internal to the inertial measurement unit (IMU) 132, or any combination thereof. Based on the one or more measurement signals from the one or more position sensors 128, the inertial measurement unit (IMU) 132 may generate fast calibration data indicating an estimated position of the near-eye display device 120. Estimated positions may be of a reference point on the near-eye display device 120, and estimated positions may be, for example, relative to an initial position of the near-eye display device 120, relative to other objects in an external environment, relative to virtual objects in an artificial environment or augmented / mixed reality, etc., as would be understood by one of ordinary skill in the art. For example, the inertial measurement unit (IMU) 132 may integrate measurement signals received from accelerometers over time to estimate a velocity vector and integrate the velocity vector over time to determine an estimated position of the near-eye display device 120. Alternatively, the inertial measurement unit (IMU) 132 may provide the sampled measurement signals to the optional console 110, which may determine the fast calibration data.

[0073] In some examples, the wireless communication subsystem 134 may include an ultra-wide band (UWB) transceiver. Ultra-wide band (UWB) wireless communication technology is used for short-range, fast, and secure data transmission environments. Ultrawide band (UWB) wireless communication technology provides high transmission speed, low power consumption, and large bandwidth, in addition to the ability to co-exist with other wireless transmission technologies. The ultra-wide band (UWB) transceiver may be used to detect another user (head-mounted display (HMD) device) within range of communication and within an angle-of-arrival (AoA), then establish line-of-sight (LoS) communication between the two users. The communication may be in audio mode or in audio / video mode. In other examples, the ultra-wide band (UWB) transceiver may be used to detect the other user, but a different communication technology (transceiver) such as WiFi or Bluetooth Low Energy (BLE) may be used to facilitate the line-of-sight (LoS) communication. In some examples, the wireless communication subsystem 134 may include one or more global navigation satellite system (GNSS) receivers, such as, e.g., a global positioning service (GPS) receiver, one or more transceivers compliant with the Institute of Electrical & Electronic Engineers (IEEE) 803.11 family of present and / or future standards (such as, e.g., "WiFi"), one or more Bluetooth transceivers, one or more cellular receivers and / ortransmitters (compliantwith any of the 3rdGeneration Partnership Project (3GPP), Open Radio Access Network (O- RAN), evolved Common Public Radio Interface (eCPRI), etc., standards), and / or any other receiver and / or transmitter compliant with any suitable communication protocol (also including any unnamed protocols, such as WiMax, NearLink, Zigbee, etc., that would be known to one of ordinary skill in the art). In some instances, any of these communication transceivers may also be implemented in other suitable components of the near-eye display device 120, I / O interface 140, and / or console 110. In some cases, multiple wireless communication transceivers may be available for, inter alia, the wireless communication subsystem 134 and / or other components of the system 100, and the one with lowest power consumption, highest communication quality (e.g., based on interfering signals), or user choice may be used. For example, the communication technology may be selected based on a lowest power consumption for a given range.

[0074] In some examples, the one or more processors 121 may be the control electronics (which may include, e.g., an operating system) forthe near-eye display device 120. The one or more processors 121 may be employed for controlling one or more of the display electronics 122, the display optics 124, the one or more locators 126, the one or more position sensors 128, the facial feature tracking unit 130, the inertial measurement unit (IMU) 132, the wireless communication sub-system 134, the one or more outward projectors 172, and / or the one or more inward projectors 173, according to the present disclosure. The one or more processors 121 may be implemented, in whole or in part, as a separate physical component in the near-eye display device 120, as distributed among and / or integrated into one or more components of the near-eye display device 120 (such as, e.g., the display electronics 122), and / or externally to near-eye display device 120, such as being implemented / integrated in, for example, the input / output interface 140 and / or the console 110 (e.g., the facial feature tracking module 118, the headset tracking module 114, the virtual reality engine 116, the application store 112, etc.), and / or in another external system connected by, for example, the wireless communication subsystem 134. In some examples, the one or more processors 121 of the near-eye display device 120 may receive input, store, and process data, and / or control the components of the near-eye display device 120 in accordance with received input and / or stored / processed data in order to maintain optimal operating conditions of one or more components in the near-eye display device 120.

[0075] In some examples, the one or more processors 121, any control electronics,and / or any of the other components of the near-eye display device 120 may be implemented in and / or by any number of processors executing instructions stored on any number of non- transitory computer-readable storage media (not shown) disposed on / in and / or communicatively linked to the near-eye display device 120. The one or more processors 121 may include multiple processing units executing instructions in parallel. The non-transitory computer-readable storage medium / media may be any memory, such as a hard disk drive, a removable memory, or a solid-state drive (e.g., flash memory or dynamic random access memory (DRAM)). In some examples, the one or more processors 121 in the near-eye display device 120 may perform one or more functions; in some examples, one or more non- transitory computer-readable storage media in the near-eye display device 120 may store instructions that, when executed by the one or more processors 121, cause the one or more processors 121 to perform any of the functions described herein and / or to control any of the components described herein. In some examples, functions such as those described below in reference to the optional console 110 (e.g., facial feature tracking, headset tracking, and the generation of virtual reality images) may be performed by the one or more processors 121 integrated with and / or wired / wirelessly connected to the near-eye display device 120.

[0076] In some examples, the input / output interface 140 may be a device that allows a user to send action requests to the optional console 110 and / or the near-eye display device 120. As used herein, an "action request" may be a request to perform a particular action. For example, an action request may be to start orto end an application or to perform a particular action within the application. The input / output interface 140 may include one or more input devices. Example input devices may include a keyboard, a mouse, a game controller, a glove, a button, a touch screen, or any other suitable device for receiving action requests and communicating the received action requests to the optional console 110. In some examples, an action request received by the input / output interface 140 may be communicated to the optional console 110 and / or the near-eye display device 120, either or both of which may perform an action corresponding to the requested action.

[0077] In some examples, the optional console 110 may provide content to the near- eye display device 120 for presentation to the user in accordance with information received from one or more of the near-eye display device 120, the input / output interface 140, and / or the external imaging device 150. For example, as shown in the example of FIG. 1, the optional console 110 may include an application store 112, a headset tracking module 114, a virtualreality engine 116, and a facial feature tracking module 118. In some examples, the optional console 110 may include different or additional modules than those described herein, and the functions described further below may be distributed among the components of the optional console 110 in a different manner than is described here (or may be distributed, in part or whole, in one or more components in the near-eye display device 120). It should be appreciated that the optional console 110 may or may not be employed, or the optional console 110 may be integrated, in whole or in part, with the input / output interface 140 and / or the near-eye display device 120, or the optional console 110 may be separate from the input / output interface 140 and / or the near-eye display device 120. In some examples, the optional console 110 may include a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor (including, for example, the application store 112).

[0078] In some examples, the application store 112 may store one or more applications for execution by one or more processors in any one or more of the optional console 110, the near-eye display device 120, the input / output interface 140, and / or the optional external imaging device 150. An application may include a group of instructions that, when executed by a processor, generates content for presentation to the user. Examples of the applications may include gaming applications, conferencing applications, video playback application, or other suitable applications.

[0079] In some examples, the virtual reality engine 116 may execute applications within the artificial reality system environment 100 and receive position / acceleration / velocity information of the near-eye display device 120, predicted future positions of the near-eye display device 120, or any combination thereof from the headset tracking module 114. In some examples, the virtual reality engine 116 may also receive estimated eye position and orientation information from the facial feature tracking module 118. Based on the received information, the virtual reality engine 116 may determine content including, e.g., virtual reality images, to provide to the near-eye display device 120 for presentation to the user.

[0080] In some examples, the facial feature tracking module 118, which may be implemented as a processor, may receive facial feature tracking data from the facial feature tracking unit 130 and determine, for example, the position of the user's eye based on the facial feature tracking data. In some examples, the position of the eye may include an eye'sorientation, location, or both relative to the near-eye display device 120 or any element thereof. Accordingly, in these examples, because the eye's axes of rotation change as a function of the eye's location in its socket, determining the eye's location in its socket may allow the facial feature tracking module 118 to more accurately determine the eye's orientation.

[0081] In some embodiments, any one or more components shown in FIG. 1 may be further broken down into sub-components and / or combined together to form larger modules, as would be understood by one of ordinary skill in the art. For example, in some examples, the near-eye display device 120 may include additional, fewer, and / or different components than shown and / or described in reference to FIG. 1. Moreover, groupings of components may work together as sub-systems within the near-eye display device 120, and / orshare / provide / transmit data and / or control information, etc., as would be understood by one of ordinary skill in the art. For example, as indicated by the dotted line box connecting / overlappingthe display electronics 122, the one or more outward-facing sensor(s) 123, the one or more outward projectors 172, the one or more inward projectors 173, and the facial feature tracking unit 130 in FIG. 1, these listed components may work together and / or may be somewhat integrated in terms of form and / or function in actual implementations of the near-eye display device 120 in FIG. 1.

[0082] In some embodiments, any one or more of the components and / or functionalities described in reference to any of the drawings / figures herein may be implemented by hardware, software, and / or any combination thereof, according to examples of the present disclosure. In some examples, the components and / or functionalities may be implemented by any type of application, program, library, script, task, service, process, and / or any type or form of executable instructions executed on hardware such as circuitry that may include digital and / or analog elements (e.g., one or more transistors, logic gates, registers, memory devices, resistive elements, conductive elements, capacitive elements, and / or the like, as would be understood by one of ordinary skill in the art). In some examples, the hardware and data processing components used to implement the various processes, operations, logic, and circuitry described in connection with the examples described herein may be implemented with a general purpose single- and / or multi-chip processor, a single- and / or multi-core processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmablelogic device, discrete gate or transistor logic, discrete hardware components, and / or any combination thereof suitable to perform the functions described herein. A general purpose processor may be any conventional processor, microprocessor, controller, microcontroller, and / or state machine. In some examples, the memory / storage may include one or more components (e.g., random access memory (RAM), read-only memory (ROM), flash or solid state memory, hard disk storage, etc.) for storing data and / or computer-executable instructions for completing and / or facilitating the processing and storage functions described herein. In some examples, the memory / storage may be volatile and / or non-volatile memory, and may include database components, object code components, script components, or any other type of information structure suitable for implementing the various activities and storage functions described herein.

[0083] FIGS. 2A and 2B illustrate a front prospective view and a back prospective view, respectively, of a near-eye display device in the form of a head-mounted display (HMD) device 200 which may be implemented with an inward-facing and / or an outward-facing projection system to which examples of the present disclosure may be applied. In some examples, the head-mounted display (HMD) device 200 may be a specific implementation of the near-eye display 120 of FIG. 1, and may be configured to operate as a virtual reality (VR) system, an augmented reality (AR) system, and / or as part of any such digital content display system that uses displays or wearables, or any combination thereof. In some examples, the headmounted display (HMD) device 200 may include a display 210, a body 220 and a head strap 230. In some examples, the head-mounted display (HMD) device 200 may include additional, fewer, and / or different components than shown and / or described in reference to FIGS. 2A- 2B.

[0084] FIG. 2A is a frontal prospective view 200A showing a front side 225, a bottom side 223, and a right side 229 of the body 220, as well as the display 210, an outward-facing camera 250, and the head strap 230 of the head-mounted display (HMD) device 200. In some examples, two or more outward-facing cameras 250 may be employed for, e.g., a stereoscopic viewing by the user by display projectors inside the head-mounted display (HMD) device 200. FIG. 2B is a bottom rear prospective view 200B showing the bottom side 223, the front side 225, and a left side 227 of the body 220, as well as the display 210 and the head strap 230 of the head-mounted display (HMD) device 200. In some examples, the head strap 230 may have an adjustable or extendible length. In particular, in some examples, theremay be a sufficient space between the body 220 and the head strap 230 of the head-mounted display (HMD) device 200 for allowing a user to mount the head-mounted display (HMD) device 200 onto the user's head. For example, the length of the head strap 230 may be adjustable to accommodate a range of user head sizes.

[0085] In some examples, the head-mounted display (HMD) device 200 (including, e.g., the display 210) in FIGS. 2A-2B may include any number of processors, display electronics, and / or display optics similar to the one or more processors 121, the display electronics 122, and the display optics 124 described in reference to FIG. 1. For example, in some examples, the outward-facing camera 250 may correspond to the outward-facing sensor(s) of the near-eye display device 120, and may be under the control of processor(s)121, of FIG. 1, and / or be operationally connected to any one or more of the display electronics122, the one or more outward projectors 172, the one or more inward projectors 173, and the facial feature tracking unit 130 as indicated by the dotted line box connecting / overlapping those components in FIG. 1. The outward-facing camera 250 in the head-mounted display (HMD) device 200 in FIGS. 2A-2B may operate similarly to the outward-facing camera(s) 320 in FIGS. 3A-3B, as discussed and described below. As mentioned above, in some examples, the head-mounted display (HMD) device 200 in FIGS. 2A-2B may include two or more outward-facing cameras rather than a single outward-facing camera 250, such as the three outward-facing cameras employed in the Quest 3™ from Meta™.

[0086] In some examples, the display electronics and display optics of the headmounted display (HMD) device 200 may display and / or facilitate the display of media or other digital content including virtual and / or augmented views of a physical, real-world environment with computer-generated elements. Examples of the media or digital content presented by the head-mounted display (HMD) device 200 may include images (e.g., two- dimensional (2D) or three-dimensional (3D) images), videos (e.g., 2D or 3D videos), audio, or any combination thereof. In some examples, the display electronics may display a three- dimensional (3D) image, e.g., using stereoscopic effects produced by two-dimensional panels, to create a subjective perception of image depth. In some examples, the display optics in the head-mounted display (HMD) device 200 may include a single optical element or any number of combinations of various optical elements, such as waveguides, gratings, optical lenses, optical couplers, mirrors, etc., as well as mechanical couplings to maintain relative spacing and orientation of the optical elements in the combination, such as are described above inreference to the display optics 124 in FIG. 1.

[0087] In some examples, the head-mounted display (HMD) device 200 in FIGS. 2A-2B may include one or more inward / outward projectors, similar to the one or more inward projectors 173 and / or one or more outward projectors 172 of FIG. 1. In some examples, the one or more inward projectors of the head-mounted display (HMD) device 200 may project an image for direct observation by the user's eye and / or project a fringe or other pattern on the eye. In some examples, the one or more outward projectors of the head-mounted display (HMD) device 200 may project a fringe or other pattern on the external environment and / or objects / surfaces within the external environment in order to, for example, perform 3- dimensional (3D) mapping of the external environment. In some examples, the one or more inward / outward projectors of the head-mounted display (HMD) device 200 may include one or more of Vertical Cavity Surface Emitting Laser (VCSEL), a liquid crystal display (LCD) and / or a light-emitting diode (LED); more specifically, the one or more inward / outward projectors of the head-mounted display (HMD) device 200 may include, e.g., one or more of a liquid crystal display (LCD), a light emitting diode (LED) or micro-light emitting diode (mLED), an organic light emitting diode (OLED), an inorganic light emitting diode (ILED), an active-matrix organic light emitting diode (AMOLED), a transparent organic light emitting diode (TLED), any other suitable light source, and / or any combination thereof. It should be appreciated that in some examples, the inward projectors of the head-mounted display (HMD) device 200 may be placed near and / or closer to a user's eye (e.g., "eye-side"). It should be appreciated that, in some instances, utilizing a back-mounted inward projector may help to reduce size or bulkiness of any housing for a display system, which may also result in a significant improvement in user experience for a user.

[0088] In some examples, the head-mounted display (HMD) device 200 may also include a facial feature tracking system, one or more locators, one or more position sensors, and an inertial measurement unit (IMU), similar to the facial feature tracking unit 130, the one or more locators 126, the one or more position sensors 128, and the inertial measurement unit (IMU) 132, respectively, described in reference to FIG. 1. In some examples, the head-mounted display (HMD) device 100 may include various other sensors, such as depth sensors, motion sensors, image sensors, light sensors, and / or the like. Some of these sensors may sense any number of structured or unstructured light patterns projected by the one or more inward / outward projectors of the head-mounted display (HMD) device200 for any number of purposes, including, e.g., sensing, facial feature tracking, and / or the creation of virtual reality (VR) content.

[0089] In some examples, the head-mounted display (HMD) device 200 may include and / or be operably connected to a virtual reality engine (not shown), similar to the virtual reality engine 116 described in reference to FIG. 1, that may execute applications within the head-mounted display (HMD) device 200 and receive depth information, position information, acceleration information, velocity information, predicted future positions, or any combination thereof of the head-mounted display (HMD) device 200 from the various sensors. In some examples, the information received by the virtual reality engine may be used for producing a signal (e.g., display instructions) to the one or more display assemblies. In some examples, the head-mounted display (HMD) device 200 may include locators (not shown), similar to the one or more locators 126 described in reference to FIG. 1, which may be located in fixed positions on the body 220 of the head-mounted display (HMD) device 200 relative to one another and relative to a reference point. Each of the locators may emit light that is detectable by an external imaging device. This may be useful for the purposes of head tracking or other movement / orientation. It should be appreciated that other elements or components may also be used in addition or in lieu of such locators.

[0090] As stated above, the head-mounted display (HMD) device 200 may include additional, fewer, and / or different components than shown and / or described in reference to FIGS. 2A-2B. In some examples, the head-mounted display (HMD) device 200 may include an input / output interface (similar to the input / output interface 140 in FIG. 1), a console (similar to the console 110 described in reference to FIG. 1), and / or a camera to capture images or videos of the user's environment to present the user with, e.g., augmented reality (AR) / virtual reality (VR) content. In some examples, the head-mounted display (HMD) device 200 may include one or more cameras to capture reflections of patterns projected by the one or more inward / outward projectors.

[0091] FIGS. 3A and 3B illustrate a perspective view 300A and a top view 300B, respectively, of a near-eye display device 300 in the form of a pair of glasses having both an inward-facing and an outward-facing projection systems to which examples of the present disclosure may be applied. In some examples, the near-eye display device 300 may be a specific implementation of the near-eye display device 120 of FIG. 1, and may be configured to operate as a virtual reality (VR) system, an augmented reality (AR) system, and / or as partof any such content system that uses displays or wearables, or any combination thereof. As shown in FIGS. 3A-3B, the near-eye display device 300 may include a frame 305, one or more outward pattern projectors 310, one or more facial feature tracking projectors 315 (which effectively operate as inward pattern projectors), an outward-facing camera(s) 320, a facial feature tracking camera(s) 325, and a display 390.

[0092] As shown in FIGS. 3A-3B, the near-eye display device 300 may include an inward-facing imaging / projection system, including the one or more facial feature tracking projectors 315 (i.e., inward pattern projectors) and the facial feature tracking camera(s) 325, and an outward-facing imaging / projection system, including the one or more outward pattern projectors 310 and the outward-facing camera(s) 320. In some examples, the inwardfacing imaging / projection system of the near-eye display device 300 may be a facial feature tracking system, where the one or more facial feature tracking projectors 315 project a pattern directly on the user's eye(s) and / or face, and the facial feature tracking camera(s) 325 captures one or more reflections of the projected pattern from the user's eye(s) and / or face, and the facial feature tracking system uses the captured reflections to track the user's eye(s) and / or face.

[0093] In some examples, the one or more facial feature tracking projectors 315 may be disposed on the temple arms of the frame 305 of the near-eye display device 300 (not shown in either FIGS. 3A or 3B), and may project one or more patterns on eye lens of the near-eye display device 300, which reflects those one or more patterns onto the user's eye 355 and / or face (i.e., a rear projection slight source). In such examples, the inner surface of the eye lens may be coated with a reflective surface, fabricated with a reflective surface, and / or covered by a metasurface or other type of nanostructure which may be suitably employed for the re-direction of the light projected by the one or more facial feature tracking projectors 315, as would be understood by one of ordinary skill in the art. In such examples, the inner surface may create the one or more patterns which are projected onto the user's eye 355 and / or face, either alone or in combination with the one or more facial feature tracking projectors 315. In other words, in some examples, the one or more facial feature tracking projectors 315 may project unstructured light, and the inner surface both reflects and / or re-directs the light onto the user's eye and / or face while also providing one or more patterns which may be used for facial feature tracking. In some examples, the one or more facial feature tracking projectors 315 may project a pattern such as, for example, a structuredimage (e.g., a fringe pattern) projected onto the eye and / or face by a micro-electromechanical system (MEMS) based scanner reflecting light from a light source (e.g., a laser).

[0094] In some examples, the one or more facial feature tracking projectors 315 may include one or more of a light emitting diode (LED) or micro-light emitting diode (mLED) or edge emitting LED, an organic light emitting diode (OLED), an inorganic light emitting diode (ILED), an active-matrix organic light emitting diode (AMOLED), a transparent organic light emitting diode (TLED), a superlu miniscent diode (SLED), another type of suitable light emitting diode, a Vertical Cavity Surface Emitting Laser (VCSEL) or other type of laser, a photonic integrated circuit (PIC) based illuminator, a liquid crystal display (LCD), a light source with a micro-electromechanical system (MEMS) based scanner, any other suitable light source, and / or any combination thereof. In any examples employing a VCSEL, the VCSEL may have one or more of a wide variety of possible VCSEL architectures, and / or fabrications, as would be understood by one of ordinary skill in the art. In such examples, the VCSEL may include a VCSEL with multiple active regions (e.g., a bipolar cascade VCSEL); a tunnel junction VCSEL; a tunable VCSEL which may employ, e.g., a micro-electromechanical system (MEMS); a wafer- bonded and / or wafer-fused VCSEL; a Vertical External Cavity Surface Emitting Laser (VECSEL); a Vertical Cavity Semiconductor Optical Amplifier (VCSOA) which may be optimized as amplifiers as opposed to oscillators; two or more Vertical Cavity Surface Emitting Lasers (VCSELs) disposed on top of one another (i.e., vertically) such that each one pumps the one on top of it (e.g., monolithically optically pumped VCSELs); any other suitable VCSEL construction, architecture, and / orfabrication, as would be understood by one of ordinary skill in the art in light of the examples of the present disclosure; and / or other constructions, architectures, and / or fabrications suitable for the present disclosure may be employed besides a VCSEL, such as— with appropriate architectural modifications, for example, an Edge- Emitting Laser (EEL), a Horizontal Cavity Surface Emitting Laser (HC-SEL), a Quantum Dot Laser (QDL), a Quantum Cascade Laser (QCL), any other form of solid state laser, and / or any light source suitable for examples according to the present disclosure, as would also be understood by one of ordinary skill in the art.

[0095] In some examples, the facial feature tracking camera(s) 325 may be an image sensor, such as a complementary metal-oxide semiconductor (CMOS) image sensor, a defocused image sensor, a light field sensor, a single photon avalanche diode (SPAD), and / or, in certain implementations, a non-imaging sensor, such as a self-mixing interferometer (SMI)sensor. In some examples, a combined VCSEL / SMI integrated circuit may be employed as both a light source and a sensor for facial feature tracking. In such an example employing a combined VCSEL / SMI integrated circuit as both a light source and a sensor for facial feature tracking, the combined VCSEL / SMI integrated circuit may be disposed inside the frame of near-eye display device 300 and point into the waveguide 393 constituting the display 390 in order to perform facial feature tracking illumination and sensing.

[0096] As shown in FIG. 3B, in some examples, the outward-facing imaging / projection system of the near-eye display device 300 may include the one or more outward pattern projectors 310, which project a pattern directly on an external environment 350 and / or one or more objects / surfaces in the external environment 350, and the outward-facing camera(s) 320, which captures one or more reflections of the projected pattern on the one or more objects / surfaces or all or part of the entire external environment 350. In some examples, such an outward-facing imaging / projection system may serve a variety of purposes, including, but not limited to, profilometry, determining surface patterns / structures of objects in the external environment 350, determining distances from the user to one or more objects / surfaces in the external environment 350, determining relative positions of one or more objects / surfaces to each other in the external environment 350, determining relative velocities of one or more objects / surfaces in the external environment 350, etc., as would be understood by one of ordinary skill in the art. In some examples, the outward-facing imaging / projection system of the near-eye display device 300 may also be employed to capture images of the external environment 350. In such examples, the captured images may be processed, for example, by a virtual reality engine to add virtual objects to the captured images or modify physical objects in the captured images, and the processed images may be displayed to the user by the display 390 for augmented reality (AR) applications.

[0097] In some examples, the display 390 may include, in whole or in part, one or more processors, display electronics, and / or display optics similar to the one or more processors 121, the display electronics 122, and the display optics 124 in FIG. 1, and may be configured to present media or other content to a user, including, e.g., virtual reality (VR), augmented reality (AR) system, and / or any other system capable of presenting media or other content to a user. In some examples, the display 390 may include any number of light sources, such as, e.g., Vertical Cavity Surface Emitting Laser (VCSEL), a liquid crystal display (LCD) display panel, a light-emitting diode (LED) display panel, or an optical display panel (e.g., awaveguide display assembly), etc., and any number of optical components, such as waveguides, gratings, lenses, mirrors, etc., as would be understood by one of ordinary skill in the art.

[0098] As shown in FIG. 3B, in some examples, the display 390 of the near-eye display device 300 may include optics 391 and a waveguide 393, which may be coupled to a projector (such as, e.g., the one or more inward projectors 173 of FIG. 1). In some examples, the projectors may be disposed inside the frame on the sides of the waveguide 393 constituting the display 390, thereby projecting light into and through the waveguide 393, which, in turn, projects the light towards the user's eye. In some examples, the display 390 may combine the view of the external environment 350 and artificial reality content (e.g., computer-generated images). In some examples, light from the external environment 350 may traverse a "see- through" region of the waveguide 393 in the display 390 to reach a user's eye 355 (located somewhere within an eye box), while images are also projected for the user to see as part of an augmented reality (AR) display.

[0099] In such examples, the light of images projected by the projector may be coupled into a transparent substrate of the waveguide 393, propagate within the waveguide 393, be coupled with light from the user's actual environment, and be directed out of the waveguide 393 at one or more locations towards a user's eye 355 located within the eye box. In such examples, the waveguide 393 may be geometric, reflective, refractive, polarized, diffractive, and / or holographic, as would be understood of one of ordinary skill in the art, and may use any one or more of macro-optics, micro-optics, and / or nano optics (such as, e.g., metalenses and / or metasurfaces). In some examples, the optics 391 of the display 390 may include optical polymers, plastic, glass, transparent wafers (e.g., Silicon Carbide (SiC) wafers), amorphous silicon, Silicon Oxide (SiO?), Silicon Nitride (SiN), Titanium Oxide (TiO), optical nylon, carbon-polymers, and / or any other transparent materials used for such a purpose, as would be understood by one of ordinary skill in the art.

[0100] In some examples, the near-eye display device 300 may further include various sensors on or within a frame 305, such as, e.g., any number of depth sensors, motion sensors, position sensors, inertial sensors, and / or ambient light sensors. In some examples, the various sensors may include any number of image sensors configured to generate image data representing different fields of views in one or more different directions (which may or may not include the outward-facing camera(s) 320). In some examples, the various sensors maybe used as input devices to control or influence the displayed content of the near-eye display device 300, and / orto provide an interactive virtual reality (VR) and / or augmented reality (AR) experience to a user of the near-eye display device 300. In some examples, the various sensors may also be used for stereoscopic imaging or other similar application.

[0101] In some examples, the near-eye display device 300 may further include one or more illuminators to project light into a physical environment (which may or may not include, e.g., the outward pattern projector(s) 310). The projected light may be associated with different frequency bands (e.g., visible light, infra-red light, ultra-violet light, etc.), and may serve various purposes. In some examples, the one or more illuminators may be used as locators, such as the one or more locators 126 described above with respect to FIG. 1. In such examples, the near-eye display device 300 may also include an image capture unit (which may or may not include the outward-facing camera(s) 320 and / or the external imaging device 150 of FIG. 1), which may capture images of the physical environment in the field of view. In some instances, the captured images may be processed, for example, by a virtual reality engine (such as, e.g., the virtual reality engine 116 of FIG. 1) to add virtual objects to the captured images or modify physical objects in the captured images, and the processed images may be displayed to the user by the display 390 for augmented reality (AR) applications.

[0102] In some examples, a majority of electronic components of the near-eye display device 300 in the form of a pair of glasses may be included in the frame 305 of the glasses (e.g., a top bar, a bridge, a rim, a lens, etc.). Examples of such electronic components included in the frame 305 include, but are not limited to, a camera, a sensor, a projector, a speaker, a battery, a microphone, and a battery management unit (BMU). In some examples, a battery management unit (BMU) may be an electronic system that may be used to manage charging and discharging of a battery (e.g., a lead acid battery). In some examples, the battery management unit (BMU) may, among other things, monitor a state of the battery, determine and report data associated with the battery, and provide environmental control(s) for the battery. In some examples, the temples 306 may be provided with a tapering profile, based on design considerations for the specific implementation. In such examples, the tapered temples may be utilized to house various electronic components. For example, in some cases, a microphone or speaker may often be placed towards a rear of a temple arm, near a user's ear, and as such, in many cases, a battery may be more likely to be placed near a front of the temple arm.

[0103] In FIG. 3B, a facial feature tracking system (such as that described in reference to facial feature tracking unit 130, the facial feature tracking module 118, and the inward projector(s) 173 of FIG. 1) may be implemented by the facial feature tracking projector(s) 315, which project structured light, such as patterns and / or other suitable lighting for performing facial feature tracking upon the user's eye 355 and / or portions of the user's face, the facial feature tracking camera(s) 325, which receive reflections of the light of the facial feature tracking projector(s) 315 from the user's eye 355 and / or portions of the user's face, and a controller (or controllers) 317, which process the reflections received by the facial feature tracking camera(s) 325 to perform facial feature tracking. In some examples, the structured light may include one or more patterns. In some examples, the projected structured light may include, for example, one or more of a statistically random pattern (such as, e.g., a pattern of dots or a pattern of speckles), an interference pattern (such as, e.g., a moire pattern or a fringe pattern), a sinusoidal pattern, a binary pattern, a multi-level pattern (such as, e.g., a multi-level grayscale pattern), a code-based pattern, a color-based pattern, and a geometrical pattern (such as, e.g., a triangular, pyramidal, or trapezoidal pattern), as would be understood by one of ordinary skill in the art. Moreoever, in various examples of the present dislosure, there may be a single projected pattern, or a multitude of patterns, or a series of related patterns, which may be projected either separately, in a time series, or simultaneously, as would be understood by one of ordinary skill in the art. In some examples, periodic patterns (such as, e.g., fringe patterns) and / or non-periodic patterns (such as, e.g., speckle patterns) may be employed.

[0104] In some examples, the controller 317 may be similar to the one or more processor(s) 121 in FIG. 1 (and thus may perform a wide variety of functions for the near-eye display device 300), other processor(s) which perform several tasks, and / or a processor(s) dedicated to performing eye tracking and / or face tracking. In some examples, the controller 317 for performing eye tracking and / orface tracking may be communicatively connected with a memory, which may be a non-transitory computer-readable storage medium storing instructions executable by the controller 317. The controller 317 may include multiple processing units, and those multiple processing units may further execute instructions in parallel. The non-transitory computer-readable storage medium may be any memory, such as a hard disk drive, a removable memory, or a solid-state drive (e.g., flash memory or dynamic random access memory (DRAM)). In various examples, the controller 317 may befurther subdivided into multiple devices (for example, the functions of the controller 317 may be separated among various components, such as a digital signal processing (DSP) chip for eye and / or face tracking analysis as well as a Main processing Unit (CPU) for controlling, e.g., the facial feature tracking projector(s) 315).II. FACIAL FEATURE TRACKING SENSOR SYSTEM WITH DISTRIBUTED COMPUTE: ON-SENSOR AND / OR NEAR-SENSOR

[0105] As mentioned above, it may be desirable to reduce the size, location, power / energy, and other characteristics / parameters of the facial feature tracking system in any near-eye display device. For instance, having a smaller form factor for the facial feature tracking system may be beneficial for any near-eye display device 100, such as, e.g., the headmounted display (HMD) device 200 in FIGS. 2A-2B and / or the near-eye display device 300 in the form of a pair of glasses in FIGS. 3A-3B, to increase its overall efficiency, while also providing an optimal user experience (UX), as would be understood by one of ordinary skill in the art. As discussed herein, examples of the present disclosure may be employed in any near- eye devices, with or without display capabilities, and / or with or without Virtual Reality (VR) and / or Augmented Reality (AR) (sometimes referred to as Mixed Reality (MR)) display capabilities.

[0106] It may be beneficial for eye-tracking and face-tracking systems (hereinafter, "facial feature tracking systems") in near-eye display devices, especially in AR / VR environments, to be capable of nearto real-time processing to provide accurate and seamless user interactions. Typical architectures, where raw data from facial feature tracking sensors is transmitted to a main processor for analysis, introduce latency due to data transfer and processing bottlenecks. Additionally, the sheer volume of high-resolution data generated by the facial feature tracking sensors may strain the bandwidth and overall capabilities of the internal data transfer system, leading to inefficiencies and potential data loss, which may compromise other systems as well as the accuracy and responsiveness of the facial feature tracking system.

[0107] According to examples of the present disclosure, a "distributed compute" approach, where some of the processing typically performed at a main processing unit may be moved to other processing components nearer to the facial feature tracking sensors, may be employed in facial feature tracking systems in near-eye AR / VR display devices. In some examples, an "on-sensor" architecture may be employed, where some of the sensor raw datais processed on the same chip, integrated circuit, module, and / or component as the sensor itself. In some examples, a "near-sensor" architecture may be employed, where some of the sensor raw data is processed on a chip, integrated circuit, module, and / or component near to the sensor within the frame of the near-eye display device. In such examples, it may be possible that any chip, integrated circuit, module, and / or component within the frame of the near-eye display device may be considered "near-sensor" if, for example, some data, image, and / or AR / VR environment processing is processed outside of the near-eye display device (such as in, e.g., either the console 110 or the I / O Interface 140 shown in FIG. 1).

[0108] In systems, apparatus, and / or methods according to the present disclosure, the "distributed compute" architecture using on-sensor / near-sensor chip, integrated circuit, module, and / or other type of processing component in the facial feature tracking system and / or an outward-facing sensor system (any system configured to sense an environment external to the system (e.g., the external environment rather than the user), such as, for example, any system including the outward-facing sensor(s) 123, the outward-facing camera(s) 250 in FIGS. 2A-2B, the outward-facing camera(s) 320 in FIGS. 3A-3B). By processing data at or near the source (the sensor), these on-sensor / near-sensor architectures may drastically reduce data transfer times and alleviate bandwidth constraints. This localized processing ensures that more pertinent information is relayed to the central system(s), optimizing both speed and efficiency. Furthermore, such distributed compute architectures may provide enhanced power efficiency, a highly desirable characteristic for a wearable or portable AR / VR device, which may thereby ensure longer than typical operational times without compromising performance.

[0109] In some examples according to the present disclosure, the implementation of an on-sensor distributed compute architecture in the facial feature tracking system may offer a direct solution to many latency and bandwidth challenges. By executing primary data processing directly on the sensor, the system may rapidly convert raw optical data into structured information, which may thereby eliminate delays associated with data transmission to a main processor. This may ensure real-time responsiveness, a highly desirable characteristic for applications demanding near-to-instantaneous feedback, such as AR / VR interfaces.

[0110] Furthermore, integrating on-sensor / near-sensor raw data preprocessing as done in examples of the present disclosure, whether through conventional computer visiontechniques or machine learning (ML)-driven algorithms, may refine the data relayed to the main processing unit. This approach may filter out extraneous data and privacy-sensitive information, allowing the main processing system to focus solely on relevant gaze information. By leveraging these localized processing techniques, systems, apparatuses, and / or methods according to the present disclosure may achieve more optimal efficiency, by ensuring accurate facial feature tracking results while also conserving computational resources and power, as well as providing pathways to improved industrial design of the neareye AR / VR display device.

[0111] One of the key advantages of a distributed compute architecture, i.e., distributed computing, especially when applied to Contextual Artificial Intelligence (CAI), is that it allows for unique trade-offs. Distributed computing enables the use of a lightweight Machine Learning (ML) model that may process sensor raw data from multiple sources on a dedicated unit (i.e., an on-sensor compute architecture and / or near-sensor compute architecture) designed for power efficiency rather than for speed. For example, an on-sensor compute architecture and / or near-sensor compute architecture may process raw data from inward-facing sensors (such as, e.g., facial feature tracking sensors), outward-facing sensors, position sensors, locator(s), biomedical sensors, Inertial Measurement Units (IMU), any of the various types of possible sensors in a near-eye AR / VR display device, or any combination thereof. As another example, any sensor integrated and / or connected with an on-sensor compute architecture and / or a near-sensor compute architecture may be able to operate and generate raw data continuously, which the on-sensor compute architecture and / or a nearsensor compute architecture pre-processes that raw data, but the on-sensor compute architecture and / or a near-sensor compute architecture may transmit only periodically and / or intermittently.

[0112] For instance, while a facial feature tracking sensor may generate a continuous stream of raw data, its on-sensor compute architecture and / or an operatively connected near-sensor compute architecture may receive and pre-process that continuous stream, but only occasionally send batches of gaze vectors to the main processing unit. In such a case, instead of storing and transferring raw images having millions of pixels directly from the facial feature tracking sensor, its on-sensor compute architecture and / or an operatively connected near-sensor compute architecture would store and transfer the pre-processed data— i.e., gaze vectors generated from the billions of pixels in the raw images. The savings in resourcessuch as power, computing capacity, intra-communication capacity within the near-eye display device, etc., may be a very desirable result for Contextual Al applications that may be operational all day.

[0113] These are just examples, as there are many more possible use instances for Contextual Al (CAI) using sensors with an on-sensor compute architecture and / or that are operatively connected to a near-sensor compute. For instance, a facial feature tracking sensor configured to sense a facial feature of a user (e.g., a user's eye, the user's tissue surrounding the user's eye, etc.) is integrated with an on-sensor compute architecture and / or operatively connected to a near-sensor compute architecture may optimize computation for all day performance, doing some inference on user attention, alertness, and mood. The on-sensor compute architecture and / or operatively connected near-sensor compute architecture may either (1) capture, store, and occasionally transfer many gaze vectors to the main processing unit (as mentioned above); or (2) it could combine small batches of gaze vectors with other sensor raw data, such as outward-facing sensors (such as, e.g., cameras) and a microphone to infer more complex datapoints, such as user activities and / or interactions with people / external objects / events such that overall activity primitives would capture the flow of the day.

[0114] In some examples according to the present disclosure, the low latency signal provided by the on-sensor compute architecture may provide opportunities for driving the display to the user directly with the gaze signal. In such examples, foveated rendering, dynamic distortion correction, localized dimming for contrast enhancement, and / or similar techniques / methodologies may be employed more efficiently and / or quickly in the user display because of the low-latency data signal from the facial feature tracking sensor.

[0115] Although eye tracking sensor systems are discussed mainly above, the distributed compute architecture using one or more on-sensor / near-sensor chip, integrated circuit, module, and / or other type of processing component in systems, apparatus, and / or methods for a near-eye display device according to the present disclosure may be used in both or either the facial feature tracking system and / or the outward-facing sensor system (any system sensing the external environment ratherthan the user, such as, for example, any system including the outward-facing sensor(s) 123, the outward-facing camera(s) 250 in FIGS. 2A-2B, the outward-facing camera(s) 320 in FIGS. 3A-3B). In examples including the outwardfacing sensor system, the outward-facing sensor system may be part of and / or employed bythe AR / VR environment system of the near-eye display device.

[0116] Below, non-limiting examples of facial feature tracking sensors with an on- sensor compute architecture are described in reference to FIGS. 4A-4B, whereas a nonlimiting example of a facial feature tracking sensor operably connected to a near-sensor compute architecture is described in reference to FIG. 5. A non-limiting example of an integrated module including an outward-facing sensor with an on-sensor compute, which is operatively connected to a facial feature tracking sensor, is described in reference to FIG. 6.

[0117] Non-limiting examples of monolithic integrated circuits including both on- sensor compute architecture and dual inward / outward-facing sensors, are described in reference to FIGS. 7-8.

[0118] A non-limiting example of an integrated module including an outward-facing sensor with an on-sensor compute, which is separate from, but operatively connected to, a network of facial feature tracking sensors, is described in reference to FIG. 9.

[0119] A non-limiting example of a near-sensor compute architecture disposed separately from, but operatively connected to, both a network of facial feature tracking sensors and a network of outward-facing sensors, is described in reference to FIG. 10. A flowchart illustrating a non-limiting example of a method for an on-sensor compute architecture and / or near-sensor compute architecture to combine and / or otherwise pre- process sensor raw data from inward-facing sensors and outward-facing sensors, is described in reference to FIG. 11. Finally, possible methods for fabricating some of the integrated circuits and modules discussed within are described.NON-LIMITING EXAMPLES OFFACIAL FEATURE TRACKING SENSORS EMPLOYING DISTRIBUTED COMPUTE: ON-SENSOR & NEAR-SENSOR

[0120] FIG. 4A is a block diagram showing a facial feature tracking sensor with on- sensor compute architecture as a single integrated module 415A, where the facial feature tracking sensor 460 is pointed towards the user's face (e.g., the user's eye 455), according to an example of the present disclosure. FIG. 4A is provided to illustrate a general explanation herein of examples of a facial feature tracking sensor 460 with on-sensor compute architecture 465, where the facial feature tracking sensor 460 is pointed towards the user's face, and omits aspects, features, and / or components not germane to a general explanation of examples of a facial feature tracking sensor 460 with on-sensor compute architecture 465, where the facial feature tracking sensor 460 is pointed towards the user's face, according tothe present disclosure, as would be understood by one of ordinary skill in the art. Accordingly, the components shown in FIG. 4A may not be shown in accurate aspect and / or ratio of relative sizes (e.g., the relative sizes, shapes, and / or locations of the facial feature tracking sensor 460, the on-sensor compute architecture 465, a frame 405 of the near-eye display device, etc., in FIG. 4A may in no way approximate the sizes, relative locations, and / or relative dimensions of those components in specific implementations and / or examples). In other words, FIG. 4A is intended to illustrate general concepts related to examples of the present disclosure, and is not intended to illustrate the sizes, proportions, relative aspects, etc., of the specific components shown in FIG. 4A, as would be understood by one of ordinary skill in the art.

[0121] FIG. 4A shows a partial cross-section of frame 405 of a near-eye display device 400, within which the integrated facial feature tracking sensor with on-sensor compute architecture (e.g., integrated module 415A) includes a facial feature tracking sensor 460 and an on-sensor compute architecture 465, where the facial feature tracking sensor 460 is pointing out of frame 405 of the near-eye display device 400, roughly in the general direction of the user's eye 455 (the directionality and relative disposition of the facial feature tracking sensor 460 would depend upon the particular implementation, the type of sensor being employed, the facial feature tracking techniques and / or digital processing methodologies being utilized, etc.). As discussed above and below, the raw data from the facial feature tracking sensor 460 may be directly pre-processed by the on-sensor compute architecture 465 within the integrated module 415A, before that pre-processed data is forwarded to the more centralized facial feature tracking main processing unit 470.

[0122] In some examples, the partial section of frame 405 in FIG. 4A may be a section of the front portion of frame 305 of near-eye display device 300 in FIGS. 3A-3B; in other examples, the partial section of frame 405 may be part of front side 225 of the HMD in FIGS. 2A-2B; in yet other examples, integrated module 415A may be disposed elsewhere within and / or upon the near-eye display device 400. In some examples, facial feature tracking sensor 460 may be facial feature tracking camera(s) 325 of FIGS. 3A-3B, and / or part of facial feature tracking unit 130 and / or facial feature tracking module 118 of FIG. 1. In some examples, the facial feature tracking main processing unit 470 may represent multiple components involved with performing facial feature tracking for the near-eye display device 400. In some examples, the facial feature tracking main processing unit 470 may be facial feature tracking unit 130and / or facial feature tracking module 118 of FIG. 1, or any other such facial feature tracking processing system.

[0123] In some examples, facial feature tracking sensor 460 of FIG. 4A may be a scanning sensor, such as a micro-electromechanical system (MEMS), or an ultrasonic sensing device. In some examples, facial feature tracking sensor 460 of FIG. 4A may be a camera, another type of image-based sensor, or a non-image-based sensor. In some examples, on- sensor compute architecture 465 of integrated module 415A may include processor 121 of FIG. 1 and / or a memory (not shown), which may be a non-transitory computer-readable storage medium (and may store instructions executable by the processor and / or the on- sensor compute). In some examples, on-sensor compute architecture 465 of integrated module 415A may perform any of the methods, functions, and / or processes described in any portion of the present disclosure by executing instructions contained on any suitable non- transitory computer-readable storage medium.

[0124] FIG. 4B is a block diagram of a facial feature tracking sensor 460 with on-sensor compute architecture 415B, where the facial feature tracking sensor 460 is pointed into a display waveguide 475 configured to propagate light to and from the facial feature tracking sensor 460 and a facial feature being tracked (e.g., user's eye 455) by near-eye display device 400, according to an example of the present disclosure. FIG. 4B is provided to illustrate a general explanation herein of examples of facial feature tracking sensor 460 with on-sensor compute architecture 465, where facial feature tracking sensor 460 is pointed into display waveguide 475 of near-eye display device 400, and omits aspects, features, and / or components not germane to a general explanation of examples of facial feature tracking sensor 460 with on-sensor compute 465, where facial feature tracking sensor 460 is pointed into display waveguide 475 of near-eye display device 400, according to the present disclosure, as would be understood by one of ordinary skill in the art. Accordingly, the components shown in FIG. 4B may not be shown in accurate aspect and / or ratio of relative sizes (e.g., the relative sizes, shapes, and / or locations of facial feature tracking sensor 460, on-sensor compute architecture 465, display waveguide 475, frame 405 of near-eye display device 400, etc., in FIG. 4B may in no way approximate the sizes, relative locations, and / or relative dimensions of those components in specific implementations and / or examples). In other words, FIG. 4B is intended to illustrate general concepts related to examples of the present disclosure, and is not intended to illustrate the sizes, proportions, relative aspects,etc., of the specific components shown in FIG. 4B, as would be understood by one of ordinary skill in the art.

[0125] FIG. 4B shows a partial cross-section of frame 405 of near-eye display device 400, within which integrated module 415B includes facial feature tracking sensor 460 and on- sensor compute architecture 465, where facial feature tracking sensor 460 is pointing into display waveguide 475 of near-eye display device 400 (the directionality and relative disposition of the facial feature tracking sensor 460 would depend upon the particular display waveguide 475 implementation, the type of sensor being employed, the facial feature tracking techniques and / or digital processing methodologies being utilized, etc.). As discussed above and below, the raw data from facial feature tracking sensor 460 may be pre-processed by the on-sensor compute architecture 465 within the integrated module 415B, before that pre-processed data is forwarded to the more centralized facial feature tracking main processing unit 470.

[0126] In some examples, the partial section of frame 405 in FIG. 4B may be a section of the front portion of frame 305 of near-eye display device 300 in FIGS. 3A-3B; in other examples, the partial section of frame 405 may be part of front side 225 of the HMD in FIGS. 2A-2B; in yet other examples, facial feature tracking sensor 460 with on-sensor compute architecture 465 (e.g., integrated module 415B) may be disposed elsewhere within and / or upon near-eye display device 400. In some examples, facial feature tracking sensor 460 may be the facial feature tracking camera(s) 325 of FIGS. 3A-3B, and / or part of the facial feature tracking unit 130 and / or the facial feature tracking module 118 of FIG. 1. In some examples, facial feature tracking main processing unit 470 may represent multiple components involved with performing facial feature tracking for near-eye display device 400. In some examples, facial feature tracking main processing unit 470 may be the facial feature tracking unit 130 and / or the facial feature tracking module 118 of FIG. 1, or any other such facial feature tracking processing system. In some examples, facial feature tracking sensor 460 of FIG. 4B may be a scanning sensor, such as a micro-electromechanical system (MEMS), a camera, another type of image-based sensor, or a non-image-based sensor. In some examples, on- sensor compute architecture 465 of integrated module 415B may include processor 121 of FIG. 1 and / or a memory (not shown), which may be a non-transitory computer-readable storage medium (and may store instructions executable by the processor 121 and / or on- sensor compute architecture 465). In some examples, on-sensor compute architecture 465 ofintegrated module 415B may perform any of the methods, functions, and / or processes described in any portion of the present disclosure by executing instructions contained on any suitable non-transitory computer-readable storage medium.

[0127] In examples where facial feature tracking sensor 460 in FIGS. 4A-4B may be a camera, the facial feature tracking sensor camera with on-sensor compute architecture 465 may integrate both photodetection and computational capabilities on the same silicon chip. Typical camera sensors, such as a complementary metal-oxide semiconductor (CMOS) or a charge-coupled device (CCD) sensor, are primarily responsible merely for converting incident light into electrical signals, which may then be offloaded to separate processors for image processing tasks. However, with the integrated module 415A, 415B of FIGS. 4A-4B, certain image processing tasks, and / or pre-image-processing tasks, may be executed directly on the sensor chip, minimizing data transfer latency, and potentially improving power efficiency.

[0128] One primary application of an integrated module 415A, 415B according to examples of the present disclosure may be advanced image processing. Advanced image processing may include tasks such as noise reduction, or high dynamic range (HDR) processing. By handling these tasks directly on the integrated module 415A, 415B according to examples of the present disclosure, the amount of raw data that may be transferred and processed by external chips (constituting the rest of the facial feature tracking system) may be reduced, leading to faster processing times and reduced power consumption. This may be especially crucial for high-resolution or high framerate sensors where the sheer volume of data may be a bottleneck.

[0129] Furthermore, integrated module 415A, 415B according to examples of the present disclosure may also facilitate machine learning (ML) acceleration. By integrating specialized hardware, such as tensor processing units or neural processing units, directly onto the module 415A, 415B, real-time ML tasks such as, for example, object detection, user identity recognition, or semantic segmentation may be executed at the edge. This may reduce the latency associated with transferring data to an external processor and may also allow for more efficient and rapid ML-based decision-making directly on near-eye display device 400.

[0130] FIG. 5 is a block diagram of a facial feature tracking sensor 560 operably connected to a near-sensor compute architecture 580, which, in turn, is connected to the facial feature tracking main processing system 570, according to an example of the present disclosure. FIG. 5 is provided to illustrate a general explanation herein of examples of facialfeature tracking sensor 560 operably connected to near-sensor compute architecture 580, and omits aspects, features, and / or components not germane to a general explanation of examples of facial feature tracking sensor 560 operably connected to near-sensor compute architecture 580, according to the present disclosure, as would be understood by one of ordinary skill in the art. Accordingly, the components shown in FIG. 5 may not be shown in accurate aspect and / or ratio of relative sizes (e.g., the relative sizes, shapes, and / or locations of facial feature tracking sensor 560, near-sensor compute architecture 580, frame 505 of near-eye display device 500, etc., facial feature tracking main processing system 570 in FIG. 5 may in no way approximate the sizes, relative locations, and / or relative dimensions of those components in specific implementations and / or examples). In other words, FIG. 5 is intended to illustrate general concepts related to examples of the present disclosure, and is not intended to illustrate the sizes, proportions, relative aspects, etc., of the specific components shown in FIG. 5, as would be understood by one of ordinary skill in the art.

[0131] FIG. 5 shows a partial cross-section of the frame 505 of a near-eye display device 500, within which facial feature tracking sensor 560 is operably connected to nearsensor compute architecture 580. In FIG. 5, near-sensor compute architecture 566 may receive and pre-process the raw data from facial feature tracking sensor 560, before forwarding the pre-processed data to the more centralized facial feature tracking main processing unit 570. Although facial feature tracking sensor 560 is shown pointing out of frame 505 towards the user in FIG. 5 (e.g., user's eye 555), facial feature tracking sensor 560 in some examples may be completely inside frame 505, directed into display waveguide 475 (e.g., facial feature tracking sensor 460 shown in FIG. 4B).

[0132] In some examples, the partial section of frame 505 shown in FIG. 5 may be a section of the front portion of frame 305 of near-eye display device 300 in FIGS. 3A-3B; in other examples, the partial section of frame 505 may be part of front side 225 of the HMD in FIGS. 2A-2B; in yet other examples, the facial feature tracking sensor 560 and / or the nearsensor compute architecture 566 may be disposed elsewhere in and / or on the near-eye display device 500. In some examples, the facial feature tracking sensor 560 may be the facial feature tracking camera(s) 325 of FIGS. 3A-3B, and / or part of the facial feature tracking unit 130 and / or the facial feature tracking module 118 of FIG. 1. In some examples, facial feature tracking main processing unit 570 may represent multiple components involved with performing facial feature tracking for near-eye display device 500. In some examples, facialfeature tracking main processing unit 570 may be facial feature tracking unit 130 and / orfacial feature tracking module 118 of FIG. 1, or any other such facial feature tracking processing system. In some examples, facial feature tracking sensor 560 of FIG. 5 may be a scanning sensor, such as a micro-electromechanical system (MEMS), or an ultrasonic sensing device. In some examples, facial feature tracking sensor 560 of FIG. 5 may be a camera, another type of image-based sensor, or a non-image-based sensor.

[0133] In some examples, near-sensor compute architecture 566 in FIG. 5 may include processor 121 as shown in FIG. 1 and / or a memory (not shown), which may be a non- transitory computer-readable storage medium (and may store instructions executable by the processor and / or the near-sensor compute architecture itself). In some examples, nearsensor compute architecture 566 of FIG. 5 may perform any of the methods, functions, and / or processes described in any portion of the present disclosure by executing instructions contained on any suitable non-transitory computer-readable storage medium.

[0134] In some embodiments, near-sensor compute architecture 566 refers to the strategic placement of low-level and / or pre-processing units adjacent to a sensor, e.g., facial feature tracking sensor 560, albeit not integrated onto the same silicon chip and / or integrated module 415B as shown in FIG. 4B. This configuration is designed to optimize data throughput and processing speed by minimizing the distance and time for data transfer between sensors and main processing unit(s) 570, and thus may be beneficial for applications where rapid data processing is essential, such as facial feature tracking in AR / VR systems in near-eye display devices 500.

[0135] In near-eye AR / VR display devices, the minimization of facial feature tracking sensor raw data latency in facial feature tracking systems may be a highly desirable characteristic. To achieve an optimal user experience ( UX), the delay between capturing real- world data and rendering the corresponding virtual output may be so small as to be imperceptible by the human user. By leveraging near-sensor compute architecture 566, raw data from sensors (e.g., optical sensors) may be processed substantially immediately after their generation as electrical signals, thereby facilitating faster overlay of virtual elements onto real-world scenes in an AR environment or ensuring real-time rendering in fully VR environment. This direct pipeline between the sensor and the adjacent near-sensor compute architecture ensures that computational overheads, typically associated with data transfer across longer distances and / or through multiple subsystems, may be significantly reduced.Also, as machine learning (ML) models become more integrated into AR / VR tasks, such as, for example, object recognition or spatial understanding, the inclusion of hardware tailored for rapid neural network inference in the near-sensor compute architecture domain may be increasingly desirable and beneficial.

[0136] In some examples, near-sensor compute architecture 566 may contain dedicated hardware / software / firmware for dimensionality reduction of the sensor raw data, thereby providing dedicated processing for the strict low-latency sensor raw data with and then much less and already-pre-processed data may be forwarded to main processing unit(s) 570, which may be thus dedicated to user experience (UX) and AR / VR applications. This may also mitigate some privacy risks.NON-LIMITING EXAMPLES OFDUAL OUTWARD-FACING SENSOR(S) & INWARD-FACING SENSOR(S) (E.G., WORLD-FACING CAMERAS & FACIAL FEATURE TRACKING SENSORS) EMPLOYING DISTRIBUTED COMPUTE: ON-SENSOR & NEAR-SENSOR

[0137] In the examples of the present disclosure in the present section, distributed compute architectures are employed to combine both the outward-facing / external environment / outside world sensor raw data with the inward-facing / user conditions / user monitoring sensor raw data to thereby increase overall efficiency, decrease latency, and / or improve the user experience (UX) of a near-eye AR / VR display device 600. In examples using on-sensor compute architecture 665, the pre-processing / low-level processing of both outward-facing sensor 680 raw data and inward-facing sensor (e.g., a facial feature tracking sensor 660) raw data within the same chip, integrated circuit, component, and / or integrated module 615 may substantially increase overall efficiency of the AR / VR main processing unit(s) 670 creating the AR / VR environment for the user. Similarly, in examples using near-sensor compute architecture 566 as shown in FIG. 5, the pre-processing / low-level processing of both outward-facing sensor 680 raw data and facial feature tracking sensor 660 raw data in a chip, integrated circuit, component, and / or integrated module 615 immediately adjacent to outward-facing sensor(s) 680 and the inward-facing sensor(s) (e.g., facial feature tracking sensor 660) may substantially increase overall efficiency of AR / VR main processing unit(s) 670 creating the AR / VR environment for the user. For instance, the pre-processing / low-level processing of data in an on-sensor compute architecture 665 and / or near-sensor compute architecture may be employed for the continual / all-day gathering of data in a Contextual Al (CAI) system. This may be important even in examples of near-eye devices 600 which do nothave a display, such as smartglasses having a facial feature tracking capability and an advanced Al assistant.

[0138] FIG. 6 is a block diagram of a near-eye device 600 including outward-facing sensor 680 with on-sensor compute architecture 665, where outward-facing sensor 680 and on-sensor compute architecture 665 are co-disposed on integrated module 615 that can be operatively connected to facial feature tracking sensor 660 pointed towards a user's face (e.g., user's eye 655), according to an example of the present disclosure. FIG. 6 is provided to illustrate a general explanation herein of examples of outward-facing sensor 680 and on- sensor compute architecture 665 co-disposed on integrated module 615 operatively connected to facial feature tracking sensor 660, and omits aspects, features, and / or components not germane to a general explanation of examples of integrated module 615 operatively connected to facial feature tracking sensor 660, according to the present disclosure, as would be understood by one of ordinary skill in the art. Accordingly, the components shown in FIG. 6 may not be shown in accurate aspect and / or ratio of relative sizes (e.g., the relative sizes, shapes, and / or locations of outward-facing sensor 680, on-sensor compute architecture 665, integrated module 615, frame 605 of near-eye display device 600, facial feature tracking sensor 660, etc., in FIG. 6 may in no way approximate the sizes, relative locations, and / or relative dimensions of those components in specific implementations and / or examples). In other words, FIG. 6 is intended to illustrate general concepts related to examples of the present disclosure, and is not intended to illustrate the sizes, proportions, relative aspects, etc., of the specific components shown in FIG. 6, as would be understood by one of ordinary skill in the art.

[0139] FIG. 6 shows a partial cross-section of frame 605 of near-eye AR / VR display device 600, within which integrated module 615 of outward-facing sensor 680 and on-sensor compute architecture 665, where outward-facing sensor 680 is pointing out of frame 605 of near-eye display device 600, towards the external environment 650 (the directionality and relative disposition of the outward-facing sensor would depend upon the particular implementation, the type of outward-facing sensor 680 being employed, the techniques and / or digital processing methodologies being utilized, etc.).

[0140] As shown in FIG. 6, integrated module 615 may be operatively connected to facial feature tracking sensor 660 such that on-sensor compute architecture 665 receives the sensor raw data of both outward-facing sensor 680 and facial feature tracking sensor 660.Although facial feature tracking sensor 660 is shown pointing out of frame 605 towards the user in FIG. 6, facial feature tracking sensor 660 in some examples may be completely inside frame 605, directed into an AR / VR display waveguide (e.g., facial feature tracking sensor 460 shown in FIG. 4B).

[0141] As discussed herein and throughout, the raw data from both outward-facing sensor 680 and facial feature tracking sensor 660 may be pre-processed by on-sensor compute architecture 656 within integrated module 615, before sending the combined pre- processed data to the more centralized AR / VR main processing system 670.

[0142] In some examples, the partial section of frame 605 shown in FIG. 6 may be a section of the front portion of corresponding frame 305 of the near-eye display device 300 in FIGS. 3A-3B; in other examples, the partial section of frame 605 may be part of the front side 225 of the HMD in FIGS. 2A-2B; in yet other examples, integrated module 615 may be disposed elsewhere within and / or upon near-eye AR / VR display device 600, such that outward-facing sensor 680 may still sense the external environment 650. In some examples, outward-facing sensor 680 may be outward-facing sensor(s) 123 of FIG. 1, outward-facing camera 250 of HMD 200 in FIGS. 2A-2B, outward-facing camera(s) 320 of FIGS. 3A-3B, and / or any other possible outward-facing sensor 680 for a near-eye AR / VR display device 600. In some examples, AR / VR main processing unit 670 may represent multiple components involved with creating an AR / VR environment for a user of a near-eye AR / VR display device 600. In some examples, AR / VR main processing unit 670 may be the processor(s) 121 and / or the virtual reality engine 116 of FIG. 1, or any other such AR / VR environment system.

[0143] In some examples, the facial feature tracking sensor 660 of FIG. 6 may be any of the facial feature tracking unit 130 and / orfacial feature tracking module 118 in FIG. 1, facial feature tracking camera(s) 325 of FIGS. 3A-3B, and / or any other suitable facial feature tracking sensor for any of the near-eye display devices as mentioned and / or described herein. In some examples, facial feature tracking sensor 660 of FIG. 6 may be a camera, another type of image-based sensor, and / or a non-image-based sensor; in other examples, facial feature tracking sensor 660 of FIG. 6 may be a scanning sensor, such as a micro-electromechanical system (MEMS), or an ultrasonic sensing device.

[0144] In some examples, on-sensor compute architecture 665 of the integrated module 615 may include one or more processors 121 as shown in FIG. 1 and / or one or more memories (not shown), which may be non-transitory computer-readable storage media (andmay store instructions executable by the processor(s) and / or on-sensor compute architecture 665). In some examples, on-sensor compute architecture 665 of integrated module 615 may perform any of the methods, functions, and / or processes described in any portion of the present disclosure by executing instructions contained on any suitable non-transitory computer-readable storage medium.

[0145] FIG. 7 is a block diagram of a monolithic integrated module 715 including both inward and outward-facing sensors and an on-sensor compute architecture, where the integrated inward-facing sensor (e.g., a facial feature tracking sensor 760) is pointed towards the user's face (e.g., the user's eye 755), according to an example of the present disclosure. FIG. 7 is provided to illustrate a general explanation herein of examples of monolithic integrated module 715 with both an inward-facing sensor (e.g., facial feature tracking sensor 760) and an outward-facing sensor 780, as well as an on-sensor compute architecture 765, and omits aspects, features, and / or components not germane to a general explanation of examples of monolithic integrated module 715 according to the present disclosure, as would be understood by one of ordinary skill in the art. Accordingly, the components shown in FIG. 7 may not be shown in accurate aspect and / or ratio of relative sizes (e.g., the relative sizes, shapes, and / or locations of outward-facing sensor 780, on-sensor compute 765, facial feature tracking sensor 760, monolithic integrated module 715, frame 705 of the near-eye display device 700, etc., in FIG. 7 may in no way approximate the sizes, relative locations, and / or relative dimensions of those components in specific implementations and / or examples). In other words, FIG. 7 is intended to illustrate general concepts related to examples of the present disclosure, and is not intended to illustrate the sizes, proportions, relative aspects, etc., of the specific components shown in FIG. 7, as would be understood by one of ordinary skill in the art.

[0146] FIG. 7 shows a partial cross-section of frame 705 of near-eye AR / VR display device 700 having monolithic integrated module 715 with an inward-facing sensor (e.g., facial feature tracking sensor 750), outward-facing sensor 780, and on-sensor compute architecture 765 disposed thereon. More specifically, monolithic integrated module 715 includes outwardfacing sensor 780, on-sensor compute architecture 765, and facial feature tracking sensor 760. The outward-facing sensor 780 may point out of frame 705 of near-eye display device 700, towards the external environment 750 (the directionality and relative disposition of the outward-facing sensor would depend upon the particular implementation, the type ofoutward-facing sensor 780 being employed, the techniques and / or digital processing methodologies being utilized, including the AR / VR environment techniques and methodologies being employed, etc.), while facial feature tracking sensor 760 is shown pointing out of frame 705 towards the user (the directionality and relative disposition of the facial feature tracking sensor would depend upon the particular implementation, the type of sensor being employed, the facial feature tracking techniques and / or digital processing methodologies being utilized, including the AR / VR environment techniques and methodologies being employed, etc.).

[0147] As shown in FIG. 7, monolithic integrated module 715 may include outwardfacing sensor 780, on-sensor compute architecture 765, and facial feature tracking sensor 760 in one single integrated module 715. Accordingly (and as discussed herein and throughout), the raw data from both outward-facing sensor 680 and facial feature tracking sensor 760 may be pre-processed by on-sensor compute architecture 765 within the monolithic integrated module 715, before sending the combined and / or pre-processed data to the more centralized AR / VR main processing system 770.

[0148] In some examples, outward-facing sensor 780 may be a sensor employed in a Simultaneous Localization and Mapping (SLAM) technique for tracking the position and orientation of the near-eye AR / VR display device 700 in real-time and / or mapping the external environment 750 in 3D. Accordingly, on-sensor compute architecture 765 may also be involved in the pre-processing for the SLAM technique using both outward-facing sensor 780 and the inward-facing sensor (e.g., facial feature tracking sensor 760) raw data.

[0149] In some examples, the partial section of frame 705 shown in FIG. 7 may be a section of the front portion of frame 305 of near-eye display device 300 in FIGS. 3A-3B; in other examples, the partial section of frame 705 may be part of front side 225 of the HMD in FIGS. 2A-2B; in yet other examples, the monolithic dual-sensing and on-compute architecture integrated module 715 may be disposed elsewhere within and / or upon the near-eye AR / VR display device 700. In some examples, outward-facing sensor 780 may be outward-facing sensor(s) 123 of FIG. 1, outward-facing camera 250 of HMD 200 in FIGS. 2A-2B, outwardfacing camera(s) 320 of FIGS. 3A-3B, and / or any other possible outward-facing sensor for near-eye AR / VR display device 700. In some examples, facial feature tracking sensor 760 of FIG. 7 may be any of facial feature tracking unit 130 and / or facial feature tracking module 118 in FIG. 1, facial feature tracking camera(s) 325 of FIGS. 3A-3B, and / or any other suitable facialfeature tracking sensorforany of the near-eye display devices as mentioned and / or described herein. In some examples, facial feature tracking sensor 760 of FIG. 7 may be a camera, another type of image-based sensor, and / or a non-image-based sensor; in other examples, facial feature tracking sensor 760 of FIG. 7 may be a scanning sensor, such as a microelectromechanical system (MEMS), or an ultrasonic sensing device. In some examples, AR / VR main processing 770 may represent multiple components involved with creating an AR / VR environment for a user of a near-eye AR / VR display device 700. In some examples, AR / VR main processing unit 770 may be processor(s) 121 and / or virtual reality engine 116 of FIG. 1, or any other such AR / VR environment system.

[0150] In some examples, on-sensor compute architecture 765 of the monolithic dualsensing on-compute architecture integrated module 715 may include one or more processors 121 as shown in FIG. 1 and / or one or more memories (not shown), which may be non- transitory computer-readable storage media (and may store instructions executable by processor(s) 121 and / or on-sensor compute 765). In some examples, on-sensor compute architecture 765 of the monolithic dual-sensing on-compute integrated module 715 may perform any of the methods, functions, and / or processes described in any portion of the present disclosure by executing instructions contained on any suitable non-transitory computer-readable storage medium.

[0151] FIG. 8 is a block diagram of a near eye device 800 having a monolithic integrated circuit 815 including an inward-facing sensor (e.g., facial feature tracking sensor 860), an outward-facing sensor 880, and an on-sensor compute architecture 865, where the integrated inward-facing sensor is pointed into a display waveguide 875 of near-eye display device 800, according to an example of the present disclosure. FIG. 8 is provided to illustrate a general explanation herein of examples of monolithic integrated circuit 815 with an inwardfacing sensor, outward-facing sensor 880, on-sensor compute architecture 865 and display waveguide 875, and omits aspects, features, and / or components not germane to a general explanation of examples of monolithic integrated circuit 815 as would be understood by one of ordinary skill in the art. Accordingly, the components shown in FIG. 8 may not be shown in accurate aspect and / or ratio of relative sizes (e.g., the relative sizes, shapes, and / or locations of the outward-facing sensor 880, on-sensor compute architecture 865, the inward-facing sensor, monolithic integrated circuit 815, frame 805 of near-eye display device 800, etc., in FIG. 8 may in no way approximate the sizes, relative locations, and / or relative dimensions ofthose components in specific implementations and / or examples). In other words, FIG. 8 is intended to illustrate general concepts related to examples of the present disclosure, and is not intended to illustrate the sizes, proportions, relative aspects, etc., of the specific components shown in FIG. 8, as would be understood by one of ordinary skill in the art.

[0152] In some embodiments, monolithic integrated module 815 in FIG. 8 is substantially the same as monolithic integrated module 715 in FIG. 7, except that, much like in the example of FIG. 4A in comparison to FIG. 4B, facial feature tracking sensor 860 in FIG. 8 is disposed within frame 805 and is pointing into display waveguide 875 of near-eye display device 800. As such, most of the description of FIG. 7 is also applicable to FIG. 8 and may not be repeated in its entirety, although some more salient points may be reiterated.

[0153] In monolithic integrated module 815 in FIG. 8, like FIG. 7, the raw data from both outward-facing sensor 880 and facial feature tracking sensor 860 may be pre-processed by on-sensor compute architecture 865 within monolithic integrated module 815, before sending the combined and / or pre-processed data to the more centralized AR / VR main processing system 870. In some examples, outward-facing sensor 880 may be a sensor employed in a Simultaneous Localization and Mapping (SLAM) technique for tracking the position and orientation of near-eye AR / VR display device 800 in real-time and / or mapping the external environment 850 in 3D. Accordingly, on-sensor compute architecture 865 may also be involved in the pre-processing for the SLAM technique using both outward-facing sensor 880 and facial feature tracking sensor 860 raw data.

[0154] In some examples, on-sensor compute architecture 865 of integrated module 815 in FIG. 8 may include one or more processors 121 as shown in FIG. 1 and / or one or more memories (not shown), which may be non-transitory computer-readable storage media (and may store instructions executable by the processor(s) and / or on-sensor compute architecture 865). In some examples, on-sensor compute architecture 865 of monolithic integrated module 815 may perform any of the methods, functions, and / or processes described in any portion of the present disclosure by executing instructions contained on any suitable non- transitory computer-readable storage medium.

[0155] FIGS. 9 and 10 apply the concepts as described and discussed herein to sensor networks: either a one-sided architecture where a sensor net is disposed directed in one direction while a single sensor pointed in the other direction having an on-compute are connected together so the sensor raw data from both directions may be pre-processed (FIG.9), or a two-sided architecture where a near compute is connected to two separate sensor nets directed in opposite directions (FIG. 10).

[0156] FIG. 9 a block diagram of a near-eye device 900 having an integrated module 915 including an outward-facing sensor 980 with an on-sensor compute 965, where integrated module 915 is operatively connected to a network of facial feature tracking sensors 961 pointed towards the user's face, according to an example of the present disclosure. FIG. 9 is provided to illustrate a general explanation herein of examples of an integrated outwardfacing sensor 980 with on-sensor compute architecture 965 co-disposed on integrated module 915 which is operatively connected to network of facial feature tracking sensors 961, and omits aspects, features, and / or components not germane to a general explanation of examples of integrated module 915 operatively connected to network of facial feature tracking sensors 961, according to the present disclosure, as would be understood by one of ordinary skill in the art. Accordingly, the components shown in FIG. 9 may not be shown in accurate aspect and / or ratio of relative sizes (e.g., the relative sizes, shapes, and / or locations of outward-facing sensor 980, on-sensor compute architecture 965, integrated module 915, frame 905 of near-eye display device 900, the relative placement, sizes, etc. of each facial feature tracking sensor 960 in network of facial feature tracking sensors 961, etc., in FIG. 9 may in no way approximate the sizes, relative locations, and / or relative dimensions of those components in specific implementations and / or examples). In other words, FIG. 9 is intended to illustrate general concepts related to examples of the present disclosure, and is not intended to illustrate the sizes, proportions, relative aspects, etc., of the specific components shown in FIG. 9, as would be understood by one of ordinary skill in the art.

[0157] FIG. 9 shows a partial cross-section of frame 905 of near-eye AR / VR display device 900 having integrated module 915 including outward-facing sensor 980 and on-sensor compute architecture 965, where outward-facing sensor 980 is pointing out of frame 905 of near-eye display device 900, towards the external environment 950 (the directionality and relative disposition of outward-facing sensor 980 would depend upon the particular implementation, the type of outward-facing sensor 980 being employed, the techniques and / or digital processing methodologies being utilized, etc.).

[0158] As shown in FIG. 9, the integrated module 915 may be operatively connected to network of facial feature tracking sensors 961 such that on-sensor compute architecture 965 receives the sensor raw data from both outward-facing sensor 980 and network of facialfeature tracking sensors 961. Each facial feature tracking sensor 960 in network of facial feature tracking sensors 961 of FIG. 9 may point out of frame 905 towards the user (e.g., user's eye 955). Additional ly, one, more, and / or all of facial feature tracking sensor(s) 960 in network of facial feature tracking sensors 961 in some examples may be completely inside frame 905, directed into an AR / VR display waveguide 875 as shown in FIG. 8.

[0159] As discussed herein and throughout, the raw data from both outward-facing sensor 980 and network of facial feature tracking sensors 961 may be directly pre-processed by on-sensor compute architecture 965 within the integrated module 915, before sending the combined pre-processed data to the more centralized AR / VR main processing system 970.

[0160] In some examples, the partial section of frame 905 shown in FIG. 9 may be a section of the front portion of frame 305 of near-eye display device 300 in FIGS. 3A-3B; in other examples, the partial section of frame 905 may be part of front side 225 of the HMD in FIGS. 2A-2B; in yet other examples, integrated module 915 may be disposed elsewhere within and / or upon near-eye AR / VR display device 900, such that outward-facing sensor 980 may still sense the external environment 950 in some fashion. In some examples, outward-facing sensor 980 may be outward-facing sensor(s) 123 of FIG. 1, outward-facing camera 250 of HMD 200 in FIGS. 2A-2B, outward-facing camera(s) 320 of FIGS. 3A-3B, and / or any other possible outward-facing sensorfor near-eye AR / VR display device 900. In some examples, AR / VR main processing unit 970 may represent multiple components involved with creating an AR / VR environment for a user of near-eye AR / VR display device 900. In some examples, AR / VR main processing unit 970 may be processor(s) 121 and / or virtual reality engine 116 of FIG. 1, or any other such AR / VR environment system.

[0161] In some examples, one or more of facial feature tracking sensor(s) 960 in network of facial feature tracking sensors 961 of FIG. 9 may be any of facial feature tracking unit 130 and / or facial feature tracking module 118 in FIG. 1, facial feature tracking camera(s) 325 of FIGS. 3A-3B, and / or any other suitable facial feature tracking sensor for any of the near-eye display devices as mentioned and / or described herein. In some examples, one or more of facial feature tracking sensor(s) 960 in network of facial feature tracking sensors 961 of FIG. 9 may be a camera, another type of image-based sensor, and / or a non-image-based sensor; in other examples, one or more of facial feature tracking sensor(s) 960 in network of facial feature tracking sensors 961 of FIG. 9 may be a scanning sensor, such as a microelectromechanical system (MEMS), or an ultrasonic sensing device.

[0162] In other examples, the sensor network and integrated modules may be reversed, i.e., there may by a network of outward-facing sensors and an integrated module including a facial feature tracking sensor with an on-sensor compute architecture. In such an example, the on-sensor compute architecture would perform similarly, i.e., the on-sensor compute architecture may receive the sensor raw data of both the facial feature tracking sensor and the network of outward-facing sensors and may directly pre-process the raw dualdirectional sensor raw data, before sending the combined pre-processed data to a more centralized AR / VR main processing system.

[0163] In some examples, on-sensor compute architecture 965 of the integrated module 915 may include one or more processors 121 as shown in FIG. 1 and / or one or more memories (not shown), which may be non-transitory computer-readable storage media (and may store instructions executable by processor(s) 121 and / or on-sensor compute architecture 965). In some examples, on-sensor compute architecture 965 of integrated module 915 may perform any of the methods, functions, and / or processes described in any portion of the present disclosure by executing instructions contained on any suitable non- transitory computer-readable storage medium.

[0164] FIG. 10 is a block diagram of a near-eye device 1000 having a distributed compute architecture where a near-sensor compute architecture 1066 is operatively connected to both a network of facial feature tracking sensors 1061 pointed towards the user's face (e.g., the user's eye 1055) and a network of outward-facing sensors pointed towards the external environment 1050, according to an example of the present disclosure. FIG. 10 is provided to illustrate a general explanation herein of examples of near-sensor compute architecture 1066 operatively connected to both an outward-facing network of sensors 1081 and an inward-facing network of sensors, and omits aspects, features, and / or components not germane to a general explanation of examples of near-sensor compute architecture 1066 operatively connected to both an outward-facing network of sensors 1081 and an inward-facing network of sensors, according to the present disclosure, as would be understood by one of ordinary skill in the art. Accordingly, the components shown in FIG. 10 may not be shown in accurate aspect and / or ratio of relative sizes (e.g., the relative sizes, shapes, and / or locations of the outward-facing sensors 1080, near-sensor compute architecture 1066, the frame 1005 of the near-eye AR / VR display device 1000, the relative placement, sizes, etc. of each of the sensors 1080, 1060 in either of the outward-facingnetwork 1081 or the inward-facing network, etc., in FIG. 10 may in no way approximate the sizes, relative locations, and / or relative dimensions of those components in specific implementations and / or examples). In other words, FIG. 10 is intended to illustrate general concepts related to examples of the present disclosure, and is not intended to illustrate the sizes, proportions, relative aspects, etc., of the specific components shown in FIG. 10, as would be understood by one of ordinary skill in the art.

[0165] FIG. 10 shows a partial cross-section of the frame 1005 of a near-eye AR / VR display device 1000 within which near-sensor compute architecture 1066 is operatively connected to a network of outward-facing sensors 1081 and a network of inward-facing sensors (e.g., facial feature tracking sensors 1060). The number, configuration, directionality, and relative disposition of each outward-facing sensor 1080 in network of outward-facing sensors 1081 would depend upon the particular implementation, the type(s) of outwardfacing sensor 1080 being employed, the techniques and / or digital processing methodologies being utilized, the AR / VR techniques and / or methodologies being employed, etc. Similarly, the number, configuration, directionality, and relative disposition of each facial feature tracking sensor 1060 in network of facial feature tracking sensors 1061 would depend upon the particular implementation, the type(s) of facial feature tracking sensor 1060 being employed, the techniques and / or digital processing methodologies being utilized (such as, e.g., specific eye tracking techniques / methodologies), the AR / VR techniques and / or methodologies being employed, etc. Although each facial feature tracking sensor 1060 in network of facial feature tracking sensors 1061 of FIG. 10 is shown pointing out of the frame 1005 towards the user (like the facial feature tracking sensor 460 shown in FIG. 4A or network of facial feature tracking sensors 961 shown in FIG. 9), one, more, and / or all facial feature tracking sensor 1060 in network of facial feature tracking sensors 1061 in some examples may be completely inside the frame, directed into an AR / VR display waveguide (e.g., facial feature tracking sensor 460 shown in FIG. 4B and facial feature tracking sensor 860 shown in FIG. 8).

[0166] In FIG. 10, the near-sensor compute architecture 1066 may receive sensor raw data from both network of outward-facing sensors 1081 and network of facial feature tracking sensors 1061 and may pre-process that combined dual direction sensor raw data, before sending the combined and / or pre-processed data to the more centralized AR / VR main processing system 1070.

[0167] In some examples, the partial section of frame 1005 shown in FIG. 10 may bea section of the front portion of the frame 305 of the near-eye display device 300 in FIGS. 3A- 3B; in other examples, the partial section of frame 1005 may be part of the front side 225 of the HMD in FIGS. 2A-2B; in yet other examples, the partial section of frame 1005 shown in FIG. 10 may represent a far wider area and a number of discrete locations on the frame 1005 of near-eye AR / VR display device 1000. For instance, the inward-facing network may include facial feature tracking sensors 1060 embedded in a display waveguide (e.g., waveguide 875 in FIG. 8), disposed in a horizontal array directed into the display waveguide, disposed on frame 1005 around an eye lens 1085, disposed on the temples of the near-eye AR / VR display device 1000 such that a scanning mirror or the like may re-direct reflections from the user's eye 1055, etc., as would be understood by one of ordinary skill in the art. Similarly, the outward-facing network 1081 may include outward-facing sensor 1080 embedded in a waveguide, disposed in a horizontal array within the frame 1005 such that outward-facing sensor 1080 may receive incoming light through the waveguide, disposed on the frame 1005 around eye lens 1085, disposed on the outside surfaces of the temples of the near-eye AR / VR display device 1000, etc., as would be understood by one of ordinary skill in the art.

[0168] In some examples, one or more of outward facing sensor 1080 in the network of outward-facing sensors 1081 may be the outward-facing sensor(s) 123 of FIG. 1, the outward-facing camera 250 of HMD 200 in FIGS. 2A-2B, the outward-facing camera(s) 320 of FIGS. 3A-3B, and / or any other possible outward-facing sensor for a near-eye AR / VR display device. In some examples, one or more of facial feature tracking sensor 1060 in network of facial feature tracking sensors 1061 of FIG. 10 may be any of the facial feature tracking unit 130 and / or facial feature tracking module 118 in FIG. 1, the facial feature tracking camera(s) 325 of FIGS. 3A-3B, and / or any other suitable facial feature tracking sensor 1060 for any of the near-eye display devices 1000 as mentioned and / or described herein. In some examples, one or more of facial feature tracking sensors 1060 in the inward-facing network (e.g., network of facial feature tracking sensors 1061) of FIG. 10 may be a camera, another type of image-based sensor, and / or a non-image-based sensor; in other examples, one or more of facial feature tracking sensors 960 in network of facial feature tracking sensors 1061 of FIG. 10 may be a scanning sensor, such as a micro-electromechanical system (MEMS), or an ultrasonic sensing device.

[0169] In some examples, the AR / VR main processing unit 1070 may represent multiple components involved with creating an AR / VR environment for a user of a near-eyeAR / VR display device 1000. In some examples, the AR / VR main processing unit 1070 may be the processor(s) 121 and / or the virtual reality engine 116 of FIG. 1, or any other such AR / VR environment system.

[0170] In some examples, the near-sensor compute architecture 1066 may include one or more processors 121 (FIG. 1) and / or one or more memories (not shown), which may be non-transitory computer-readable storage media (and may store instructions executable by the processor(s) 121 and / or the near-sensor compute architecture 1066). In some examples, the near-sensor compute architecture 1066 may perform any of the methods, functions, and / or processes described in any portion of the present disclosure by executing instructions contained on any suitable non-transitory computer-readable storage medium.

[0171] As shown in FIG. 10, the strategic placement of a near-sensor compute architecture 1066 may position low-level and / or pre-processing units adjacent to the sensor networks (and / or data busses), without being integrated into the sensor networks themselves. This configuration may optimize data throughput and processing speed by minimizing the distance and time for data transfer between the sensor nets and the AR / VR main processing unit(s) 1070, and thus may be beneficial for applications where rapid data processing is essential, such as both facial feature tracking and Simultaneous Localization and Mapping (SLAM) techniques in near-eye AR / VR display devices 1000.

[0172] In near-eye AR / VR display devices 1000, the minimization of sensor raw data latency in both facial feature tracking and SLAM components / processing units may be a highly desirable characteristic. To achieve an optimal user experience (UX), it may be beneficial to reduce the delay between capturing real-world data (by both outward-facing sensor(s) 1080 and the inward-facing sensors, such as, e.g., facial feature tracking sensor(s) 1060) and rendering the corresponding virtual output and / or performing near-real-time SLAM processing. By leveraging near-sensor compute architecture 1066, raw data from both facial feature tracking sensor(s) 1060 and outward-facing sensor(s) 1080 may be processed substantially immediately, facilitating faster overlay of virtual elements onto real-world scenes and / or ensuring real-time rendering in fully virtual environments. This direct pipeline between both facial feature tracking sensor(s) 1060 and outward-facing sensor(s) 1080 and the adjacent processor (i.e., near-sensor compute architecture 1066) ensures that computational overheads, typically associated with data transfer across longer distances or through multiple subsystems, may be significantly reduced. Also, as machine learning (ML)models become more integrated into near-eye AR / VR display devices 1000 for tasks like object recognition or spatial understanding, the inclusion of hardware tailored for rapid neural network inference in the near-sensor domain may be increasingly desirable.

[0173] In some embodiments, using distributed compute architectures with on- sensor compute architecture and / or near-sensor compute architecture in inward-facing and / or outward-facing sensor(s) such as described and discussed above in reference to FIGS. 4A-10 may offer several advantages:• Latency Reduction: On-sensor compute architecture may swiftly process raw optical data, reducing the time to relay information to the main processing chip. This ensures, e.g., real-time pupil position capture, crucial for accurate gaze determination in AR applications.• Data Efficiency: On-sensor processing can preprocess and filter relevant eye movement patterns, forwarding more pertinent data. This aids in efficient ML model inference, optimizing the workload of the main processing chip.• Power Optimization: Localized processing on and / or near the sensor reduces data transfer, leading to decreased power consumption. This is vital for wearable AR / VR devices where battery life and thermal management may be an issue.• Heat distribution: distributing processing permits distributing the heat over multiple spots device, reducing risk of overheating.• Enhanced Parallelism: With both on-sensor compute architecture (and / or nearcompute) and a main processor, tasks can be distributed and executed concurrently. This parallelism ensures the near-eye AR / VR display device remains responsive even under heavy computational demands.• Contextual Artificial Intelligence (CAI): the use of on-sensor and near-sensor computes (i.e., distributed computing) may enable the use of a lightweight Machine Learning (ML) where may the on-sensor compute architecture and / or near-sensor compute architecture process sensor raw data from multiple sources in a distributed architecture designed for power efficiency rather than for speed.• Robust Feedback Loops: Immediate processing on and / or near the sensor allows for rapid feedback to the ML models. This may enhance real-time adaptive sampling algorithms.• System Integration and Compactness: Integrating both processing units can lead to a more compact design, reducing the footprint of, for example, the eye-tracking module, a highly desirable characteristic in the ergonomic design of AR / VR wearables.NON-LIMITING EXAMPLE OFMETHOD FOR COMBINING / PRE-PROCESSING BOTH INWARD-FACING AND OUTWARD-FACING SENSOR RAW DATAIN A NEAR-EYE AR / VR DISPLAY DEVICE

[0174] FIG. 11 is a flowchart illustrating a method for an on-sensor compute architecture and / or near-sensor compute architecture to combine and / or otherwise pre- process sensor raw data from both one or more inward-facing sensors and one or more outward-facing sensors to optimize efficient power conversion and gaze vectors storage, according to an example of the present disclosure. The method 1100 shown in FIG. 11 is provided by way of example and may be one part of an entire process, procedure, ongoing operation, method, etc., as would be understood by one of ordinary skill in the art. The method 1100 may further omit parts of any process, procedure, ongoing operation, method, etc., involved in combining / pre-processing sensor raw data not germane to examples of the present disclosure, as would be understood by one of ordinary skill in the art. Each block shown in FIG. 11 may further represent one or more steps, processes, methods, or subroutines, as would be understood by one of ordinary skill in the art. For the sake of convenience and ease of explanation, the blocks in FIG. 11 may refer to the components shown in the FIGS, described herein; however, the method 1100 is not limited in any way to the components, apparatuses, and / or constructions described and / or shown in any of the FIGS, herein.

[0175] Some of the processes indicated by the blocks in FIG. 11 may overlap, occur substantially simultaneously, and / or be continually repeated, and, moreover, the blocks may be performed by different processing components. In some examples, the components performing any of blocks in FIG. 11 may include any one or more of the on-sensor compute architecture in FIGS. 4A, 4B, 6, 7, 8, and / or 9; the near-sensor compute architecture of FIG. 5; and / or the near-sensor compute architecture 1000 of FIG. 1000, and / or any other suitable processor / controller, as would be understood by one of ordinary skill in the art.

[0176] At block 1110, a facial feature tracking sensor, an outward-facing sensor, or any other suitable device-mounted sensor can be configured to sense and record sensor rawdata. For example, the facial feature tracking sensor can be configured to sense the user's eye, tissue surrounding the user's eye, the user's eyebrows, orthe like. Likewise, the outwardfacing sensor can be configured to sense an environment external to the user and / or the neareye device.

[0177] At block 1120, an on-sensor compute architecture and / or near-sensor compute architecture can receive sensor raw data from one or more inward-facing sensors. In some examples, the one or more inward-facing sensors may be any of the facial feature tracking sensors in any of FIGS. 4A-10. In some examples, the one or more facial feature tracking sensors may directly face the user's eye (e.g., FIGS. 4A, 5, 6, 7, 9 and / or 10); in other examples, the one or more facial feature tracking sensors may face into a display waveguide (e.g., FIGS. 4B and 8). In some examples, the one or more facial feature tracking sensors may include a multitude of facial feature tracking sensors in any configuration, disposition, and / or orientation (e.g., FIGS. 9 and 10).

[0178] Optionally, at block 1120, the on-sensor compute architecture and / or nearsensor compute architecture receives sensor raw data from one or more outward-facing sensors. In some examples, the one or more outward-facing sensors may be any of the outward-facing sensors in any of FIGS. 6-10. In some examples, the one or more outwardfacing sensors may be cameras and / or other image sensors; in other examples, the one or more outward-facing sensors may be non-image sensors.

[0179] At block 1130, the on-sensor compute architecture and / or near-sensor compute architecture can combine and / or pre-process both the sensor raw data received from the one or more inward-facing sensors and the sensor raw data received from the one or more outward-facing sensors. In some embodiments, what and how the on-sensor compute architecture and / or near-sensor compute architecture combines / preprocesses the sensor raw data would depend upon the strategy for spatial and / or temporal sampling, which may be uniform or non-uniform— a primary goal of which may be the optimization of power consumption, the reductions of system resources, etc. In some examples, bursts of dense sampling may be triggered when the user attempts to access sensitive / private data through an Al assistant and / or another interface, because user authentication may be performed because user authentication. Such user authentication may include saccadic-motion based authentication, iris authentication, periocular region-based authentication, etc., as would be known by one of ordinary skill in the art. In some examples, the on-sensor computearchitecture and / or near-sensor compute architecture of a facial feature tracking sensor may convert the sensor raw data to gaze vectors, pupil position in three-dimensional space, and / or pupil size estimation.

[0180] At block 1140, the on-sensor compute architecture and / or near-sensor compute architecture can transmit the combined / pre-processed sensor raw data to the AR / VR main processing unit.NON-LIMITING EXAMPLE OF FABRICATION METHODS

[0181] In some examples, an integrated on-sensor module (such as, e.g., the integrated facial feature tracking sensor / on-sensor compute architecture 415A in FIG. 4A; the integrated facial feature tracking sensor / on-sensor compute architecture 415B in FIG. 4B; the integrated outward-facing sensor / on-sensor compute architecture 600 in FIG. 6; the integrated outward-facing sensor / on-sensor compute architecture 900 in FIG. 9; and / or any similar on-sensor compute architecture integrated circuit in accordance with the present disclosure) and / or a monolithic dual-facing sensor with on-sensor compute architecture integrated circuit (such as, e.g., the monolithic dual inward / outward-facing sensors and on- sensor compute architecture 700 in FIG. 7; the monolithic dual eyeward-directed / outward- facing sensors and on-sensor compute architecture 800 in FIG. 8; and / or any similar monolithic dual-direction sensing / on-sensor compute architecture integrated circuit in accordance with the present disclosure) may be fabricated by a lithographic technique or process.

[0182] In such examples, the lithographic technique / process may include, for example, such techniques as photolithography (including, e.g., optical lithography and quantum optical lithography), scanning lithography (including, e.g., electron-beam lithography, scanning probe lithography, proton beam writing, charged particle lithography, etc.), soft lithography (including, e.g., polydimethylsiloxance (PDMS) lithography, microcontact printing, multilayer soft lithography, etc.), nanoimprint lithography, magnetolithography, nanofountain drawing, nanosphere lithography, neural particle lithography, plasmonic lithography, stencil lithography, and / or any other past, present, or future lithographic technique suitable for fabricating an integrated circuit, optical module, and / or chip in accordance with the present disclosure, as would be understood by one of ordinary skill in the art.

[0183] In some examples, semiconductor and / or dielectric layers may be epitaxiallydeposited on a substrate, such as Gallium Arsenide (GaAs), Aluminum Arsenide (AlAs), and / or any of the various types of silicon (e.g., Silicon Dioxide (SiOz), Silicon Nitride (Si N ), etc.), as would be understood by one of ordinary skill in the art. In some examples, one or more layers may constitute a Vertical Cavity Surface Emitting Laser (VCSEL), which may include reflector layers (such as, e.g., a distributed Bragg reflector (DBR) layer) and an active region / optical cavity layer (such as, e.g., a photon absorption layer, such as, e.g., an Indium Gallium Arsenide (InGaAs) layer, a resonant optical cavity, one or more quantum wells, in single quantum well or multiple quantum well (MQW) structures, and / or an aperture).

[0184] In some examples, beam-forming element(s) and / or any other suitable components for light modification may be fabricated, such as, for example, refractive elements, reflective elements, polarization elements, a Pancharatnam-Berry phase (PBP) or other phase-modification elements, diffractive gratings (such as, e.g. Polarization Volumetric Hologram-based (PVH) gratings, Surface Relief Gratings (SRGs), Volume Bragg Gratings (VBGs), a diffractive optical element (DOE), etc.), nano-optics (including, e.g., metalenses and metasurfaces), micro-structures (including those fabricated using 3D printing), surface coatings, lithographically-created layered waveguides, and / or any other suitable technique, technology, layer, coating, and / or material feasible and / or possible either presently or in the future, as would be understood by one of ordinary skill in the art. In some examples, electrical contacts may be included in any of the layers for purposes of control, management, and power of components / layers of the integrated circuit or chip. In some examples, a top surface may include, e.g., an emitting surface layer, a passivation layer, a surface grating (such as, e.g., a diffractive grating, relief grating, high-contrast grating, etc.), a metasurface and / or metalens, a micro-electromechanical system (MEMS), a liquid lens, a mask (such as, e.g., a phase mask), etc., as would be understood by one of ordinary skill in the art. In some examples, the top layer may be a mask suited for projecting light from a self-mixing interferometer (SMI) facial feature tracking sensor.

[0185] In some examples, the fabrication method may employ any of the various techniques of wafer processing, die preparation, packaging, and / or testing. In some examples, such wafer processing techniques may include wet cleans (including, e.g., wafer scrubbing and / or cleaning by solvents and / or solutions); surface passivation; ion implantation; molecular beam epitaxy (MBE); plasma ashing; thermal treatments (such as, e.g., rapid thermal anneal, furnace anneals, thermal oxidation, etc.); ElectrochemicalDeposition (ECD) and / or electroplating; Chemical Vapor Deposition (CVD); Atomic Layer Deposition (ALD); Physical Vapor Deposition (PVD) (including, e.g., sputtering, evaporation, etc.); Chemical Mechanical Polishing (CMP); photolithographic techniques (such as, e.g., photoresist coating, photoresist baking, edge bead removal, exposure, development, Post Exposure Baking (PEB), etc.); etching or microfabrication (such as, e.g., dry or plasma etching, including Reactive Ion Etching (RIE) and Atomic Layer Etching (ALE), and / or wet etching, including, e.g., a buffered oxide etch); laser lift-off; wafer testing, etc., as would be understood by one of ordinary skill in the art.

[0186] In some examples, the die preparation techniques may include through-silicon via (TSV), wafer mounting with dicing tape, wafer backgrinding and polishing, wafer bonding and stacking, redistribution layer manufacture, wafer bumping, die cutting, wafer dicing, etc., as would be understood by one of ordinary skill in the art. In some examples, integrated circuit packaging techniques may be employed, including, but not limited to, die attachment, bonding (such as, e.g., wire bonding, thermosonic bonding, flip chip or Tape Automated Bonding (TAB)), encapsulation (such as, e.g., integrated heat spreader (HIS) installation, molding, baking, electroplating, laser marking, silkscreen printing, trimming and forming, and the like), etc., as would be understood by one of ordinary skill in the art.

[0187] According to examples, systems and apparatuses employing a distributed compute architecture in a near-eye augmented reality (AR) / virtual reality (VR) display device using an on-sensor compute architecture and / or a near-sensor compute architecture are described herein. One or more near-eye AR / VR display devices using an on-sensor compute architecture and / or a near-sensor compute architecture are described herein. One or more methods for facial feature tracking in a near-eye device using an on-sensor compute architecture and / or a near-sensor compute architecture are described herein. One or more methods for AR / VR pre-processing in a near-eye AR / VR display device using an on-sensor compute architecture and / or a near-sensor compute architecture are also described herein. One or more methods and / or systems for manufacturing an integrated circuit with an on- sensor compute architecture and / or a near-sensor compute architecture for a near-eye AR / VR display device are also described herein. A non-transitory computer-readable storage medium may have an executable stored thereon, which when executed instructs a processor to perform any of the methods described herein.

[0188] As discussed above, any of the modules, components, integrated circuits, on-sensor computes, and / or near-sensor computes discussed and / or described herein may include one or more processors and one or more non-transitory computer-readable storage media storing instructions executable on such processors and / or any of the modules, components, integrated circuits, on-sensor computes, and / or near-sensor computes discussed and / or described herein. In some examples, such processors and / or any of the modules, components, integrated circuits, on-sensor computes, and / or near-sensor computes discussed and / or described herein may be implemented as hardware, software, and / or a combination of hardware and software in the near-eye display device. In some examples, such processors and / or any of the modules, components, integrated circuits, on- sensor computes, and / or near-sensor computes discussed and / or described herein may be implemented, in whole or in part, by any type of application, program, library, script, task, service, process, and / or any type or form of executable instructions executed on hardware such as circuitry that may include digital and / or analog elements (e.g., one or more transistors, logic gates, registers, memory devices, resistive elements, conductive elements, capacitive elements, and / or the like, as would be understood by one of ordinary skill in the art). In some examples, such processors and / or any of the modules, components, integrated circuits, on-sensor computes, and / or near-sensor computes discussed and / or described herein may be implemented with a general purpose single- and / or multi-chip processor, a single- and / or multi-core processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and / or any combination thereof suitable to perform the functions described herein. A general purpose processor may be any conventional processor, microprocessor, controller, microcontroller, and / or state machine. In some examples, the one or more non-transitory computer-readable storage media may be implemented by one or more components (e.g., random access memory (RAM), read-only memory (ROM), flash or solid state memory, hard disk storage, etc.) for storing data and / or computer-executable instructions for completing and / or facilitating the processing and storage functions described herein. In such examples, the one or more non-transitory computer-readable storage media may be volatile and / or non-volatile memory, and may include database components, object code components, script components, or any other type of information structure suitable for implementing the various activities and storage functions described herein.

[0189] As would be understood by one of ordinary skill in the art, in some embodiments, any one or more of the components and / or functionalities described in reference to any of the FIGS, herein may be implemented by hardware, software, and / or any combination thereof, according to examples of the present disclosure. In some examples, the components and / or functionalities may be implemented by any type of application, program, library, script, task, service, process, and / or any type or form of executable instructions stored in a non-transitory computer-readable storage medium executed on hardware such as circuitry that may include digital and / or analog elements (e.g., one or more transistors, logic gates, registers, memory devices, resistive elements, conductive elements, capacitive elements, and / or the like, as would be understood by one of ordinary skill in the art). In some examples, the hardware and data processing components used to implement the various processes, operations, logic, and circuitry described in connection with the examples described herein may be implemented with one or more of a general purpose single- and / or multi-chip processor, a single- and / or multi-core processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and / or any combination thereof suitable to perform the functions described herein. A general purpose processor may be any conventional processor, microprocessor, controller, microcontroller, and / or state machine. In some examples, the memory / storage may include one or more components (e.g., random access memory (RAM), read-only memory (ROM), flash or solid state memory, hard disk storage, etc.) for storing data and / or computer-executable instructions for completing and / or facilitating the processing and storage functions described herein. In some examples, the memory / storage may be volatile and / or non-volatile memory, and may include database components, object code components, script components, or any other type of information structure suitable for implementing the various activities and storage functions described herein.

[0190] In the foregoing description, various examples are described, including devices, systems, methods, and the like. For the purposes of explanation, specific details are set forth in order to provide a thorough understanding of examples of the disclosure. However, it will be apparent that various examples may be practiced without these specific details. For example, devices, systems, structures, assemblies, methods, and other components may be shown as components in block diagram form in order not to obscure theexamples in unnecessary detail. In other instances, well-known devices, processes, systems, structures, and techniques may be shown without necessary detail in orderto avoid obscuring the examples.

[0191] The figures / drawings and description are not intended to be restrictive. The terms and expressions that have been employed in this disclosure are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof. The word "example" is used herein to mean "serving as an example, instance, or illustration." Any embodiment, implementation, construction, architecture, method, process, and / or design described herein as "example1is not necessarily to be construed as preferred or advantageous over any other embodiment, implementation, construction, architecture, method, process, and / or design.

[0192] Although the methods and systems as described herein may be directed mainly to digital content, such as videos or interactive media, it should be appreciated that the methods and systems as described herein may be used for other types of content or scenarios as well. Other applications or uses of the methods and systems as described herein may also include social networking, marketing, content-based recommendation engines, and / or other types of knowledge or data-driven systems.

Claims

CLAIMS1. A system, comprising: an integrated circuit in a near-eye device; a facial feature tracking sensor configured to sense a user's eye or surrounding facial features from the near-eye device; and an on-sensor compute architecture configured to receive sensor raw data from the facial feature tracking sensor, pre-process the sensor raw data, and transmit pre-processed facial feature tracking sensor data to a facial feature tracking main processing unit.

2. The system of claim 1, wherein the facial feature tracking sensor is directed to at least one of a user's eye, facial features surrounding the user's eye, or a waveguide configured to propagate light to and from the facial feature tracking sensor and a facial feature being tracked.

3. The system of claim 1 or 2, wherein the on-sensor compute architecture is configured to optimize efficient power conversion and gaze vectors storage.

4. The system of any one of the preceding claims, further comprising a near-sensor compute architecture disposed separately from the on-sensor compute architecture; preferably wherein the on-sensor compute architecture and the near-sensor compute architecture comprise a distributed compute architecture; and preferably wherein the distributed compute architecture comprises employing the on-sensor compute architecture, the near-sensor compute architecture, or both the on-sensor compute architecture and the near-sensor compute architecture as at least a portion of a contextual artificial intelligence (CAI) system.

5. The system of claim 4, further comprising an outward-facing sensor configured to sense an environment external to the system; preferably wherein the outward-facing sensor is configured to perform within the distributed compute architecture.

6. The system of any one of the preceding claims, wherein the near-eye device comprises a frame, wherein the facial feature tracking sensor is disposed in the frame and the facial feature tracking main processing unit is disposed in the frame; and preferably wherein the near-sensor compute architecture is disposed in the frame.

7. The system of any one of the preceding claims, wherein the on-sensor architecture comprises disposing the integrated circuit or a module together with the facial feature tracking sensor on a chip.

8. A system, comprising: a near-eye augmented reality display device, comprising: a frame; a facial feature tracking sensor disposed in the frame configured to sense a user's facial feature and produce facial feature tracking sensor raw data; an integrated circuit disposed separately in the frame and operatively connected to the facial feature tracking sensor, comprising: an outward-facing sensor configured to sense an external environment and produce outward-facing sensor raw data; and an on-sensor compute architecture disposed on the facial feature tracking sensor configured to: receive the outward-facing sensor raw data; receive the facial feature tracking sensor raw data; pre-process both the outward-facing sensor raw data and the facial feature tracking sensor raw data; and transmit the pre-processed outward-facing and facial feature tracking sensor raw data; and an augmented reality main processing unit configured to receive the pre- processed outward-facing and facial feature tracking sensor raw data and to perform augmented reality processing.

9. The system of claim 8, wherein the facial feature tracking sensor is directed to at least one of a user's eye, facial features surrounding the user's eye, or a waveguide configured to propagate light to and from the facial feature tracking sensor and a facial feature being tracked.

10. The system of claim 8 or 9, wherein the on-sensor compute architecture is configured to optimize efficient power conversion and gaze vectors storage.

11. The system of any one of claims 8 to 10, further comprising a plurality of facial feature tracking sensors disposed in the frame configured to sense a user's eye or surrounding facial tissue and produce raw facial feature tracking sensor raw data; and / or the system, further comprises a plurality of outward-facing sensors disposed in the frame to sense the external environment and to produce raw outward-facing sensor raw data.

12. The system of any one of claims 8 to 11, further comprising a near-sensor computearchitecture disposed separately within the frame and operatively connected to the facial feature tracking sensor and the outward-facing sensor, wherein the near-sensor compute architecture is configured to: receive the outward-facing sensor raw data from the outward-facing sensor; receive the facial feature tracking sensor raw data from the facial feature tracking sensor; pre-process both the outward-facing sensor raw data and the facial feature tracking sensor raw data; and transmit the pre-processed outward-facing sensor raw data and facial feature tracking sensor raw data to the augmented reality main processing unit.

13. A method, comprising: sensing facial feature raw data from at least one facial feature tracking sensor disposed on a chip in a frame of a near-eye display device; receiving the facial feature tracking sensor raw data from the at least one facial feature tracking sensor; pre-processing the facial feature tracking sensor raw data using an integrated circuit co-disposed on the chip to provide pre- processed facial feature tracking sensor raw data; and transmitting the pre-processed facial feature tracking sensor raw data to a main processing unit operatively coupled to the chip.

14. The method of claim 13, further comprising receiving outward-facing sensor raw data from at least one outward-facing sensor disposed on the frame of the near-eye display device.

15. The method of claim 13 or 14, further comprising receiving pre-processed data from a near-sensor compute architecture disposed in the frame of the near-eye display device.

Images