Head-mounted display device

Abstract

Methods, systems, and devices for head-mounted display devices are described. A device can capture a set of images over a set of orientations using a set of cameras located on an outward-facing surface of the device. The set of images includes a first subset of images captured by a first camera and a second subset of images captured by a second camera. The device can detect a set of facial features in each of the first subset of images and the second subset of images; and measure a set of interpupillary distances over the set of orientations based on the set of facial features in each of the first subset of images and the second subset of images. The device can determine an interpupillary distance parameter based on aggregating the set of interpupillary distances over the set of orientations. The device can calibrate based on the interpupillary distance parameter.

Description

[0001] Cross-references

[0002] This patent application claims the benefit of U.S. Patent Application No. 17 / 183,241, filed February 23, 2021, entitled “HEADWEARABLE DISPLAY DEVICES”, by CHANDRASEKHAR et al., which has been assigned to the assignee of this application and whose entire contents are expressly incorporated herein by reference. Background Technology

[0003] The following content relates to head-mounted display devices, including head-mounted display devices (also known as head-mounted displays).

[0004] Systems are widely deployed to deliver various types of content, such as voice, video, packet data, messaging, and broadcasting. These systems may be capable of processing, storing, generating, manipulating, and representing information. Examples of such systems include virtual reality systems, which may include rendering hardware (e.g., personal computers) and display hardware (e.g., head-mounted displays) that support the processing of digital or virtual image information and provide stereoscopic multidimensional visualization. Some examples of virtual reality systems can support fully immersive, non-immersive, or collaborative virtual reality experiences. Other examples include entertainment systems, productivity systems, navigation systems, security and safety systems, and health and fitness systems. The quality of these different experiences can be affected by interpupillary distance, which can cause distortion and lead to a decrease in visual quality. Summary of the Invention

[0005] The described technology relates to improved methods, systems, devices, and apparatuses for supporting devices such as handheld devices, wearable devices, and head-mounted devices. Generally, the described technology provides the use of stereoscopic cameras from devices such as augmented reality head-mounted devices to correct lens distortion based on interpupillary distance (IPD). The described technology can use one or more outward-facing cameras to determine the IPD of a user wearing a head-mounted display. The head-mounted display can be configured to trigger an IPD calculation state, or can be triggered via another device (e.g., a smartphone), thereby switching the tracking camera to streaming mode. The head-mounted display can detect the user's eyes in tracking camera frames, determine the IPD using stereoscopic images captured from the head-mounted display, and refine the IPD estimate across multiple frames captured from multiple directions (e.g., angles). Based on the IPD, the head-mounted display can determine optimal lens distortion correction parameters for the IPD to improve the visual quality of virtual objects rendered via the head-mounted display.

[0006] A method for distortion correction at a device is described. The method may include: capturing a set of images on an orientation set using a camera set of the device, the image set including a first subset of images captured by a first camera in the camera set and a second subset of images captured by a second camera in the camera set; detecting a set of facial features in each of the first and second image subsets; measuring a set of interpupillary distances on the orientation set based on the set of facial features in each of the first and second image subsets; determining an interpupillary distance parameter for the device based on aggregating the interpupillary distance set on the orientation set; and calibrating the device based on the interpupillary distance parameter.

[0007] An apparatus for distortion correction is described. The apparatus may include: a processor; a memory coupled to the processor; and instructions stored in the memory. The instructions are executable by the processor to cause the apparatus to: capture a set of images on an orientation set using a camera set of the apparatus, the image set including a first subset of images captured by a first camera in the camera set and a second subset of images captured by a second camera in the camera set; detect a set of facial features in each of the first and second image subsets; measure a set of interpupillary distances on the orientation set based on the set of facial features in each of the first and second image subsets; determine an interpupillary distance parameter for the apparatus based on aggregating the interpupillary distance set on the orientation set; and calibrate the apparatus based on the interpupillary distance parameter.

[0008] Another apparatus for distortion correction is described. The apparatus may include: a unit for capturing a set of images on an orientation set using a camera set of the apparatus, the image set including a first subset of images captured by a first camera in the camera set and a second subset of images captured by a second camera in the camera set; a unit for detecting a set of facial features in each of the first and second image subsets; a unit for measuring a set of interpupillary distances on the orientation set based on the set of facial features in each of the first and second image subsets; a unit for determining an interpupillary distance parameter for the apparatus based on aggregating the interpupillary distance set on the orientation set; and a unit for calibrating the apparatus based on the interpupillary distance parameter.

[0009] A non-transitory computer-readable medium is described, storing code for distortion correction at a device. The code may include instructions executable by a processor to: capture a set of images on an orientation set using a set of cameras of the device, the set of images including a first subset of images captured by a first camera in the set of cameras and a second subset of images captured by a second camera in the set of cameras; detect a set of facial features in each of the first and second image subsets; measure a set of interpupillary distances on the orientation set based on the set of facial features in each of the first and second image subsets; determine an interpupillary distance parameter for the device based on aggregating the set of interpupillary distances on the orientation set; and calibrate the device based on the interpupillary distance parameter.

[0010] Some examples of the methods, apparatuses, or non-transitory computer-readable media described herein may also include operations, features, units, or instructions for: determining conditions for performing interpupillary distance measurements at the device; and enabling an interpupillary measurement state of the device based on said conditions. In some examples of the methods, apparatuses, or non-transitory computer-readable media described herein, measuring the set of interpupillary distances may be based on enabling the interpupillary measurement state.

[0011] Some examples of the methods, apparatuses, and non-transitory computer-readable media described herein may also include operations, features, units, or instructions for receiving a request from the device. In some examples of the methods, apparatuses, and non-transitory computer-readable media described herein, determining the conditions for performing the interpupillary measurement at the device may be based on receiving the request from the device.

[0012] Some examples of the methods, apparatuses, and non-transitory computer-readable media described herein may also include operations, features, units, or instructions for: detecting one or more signals based on one or more sensors of the device; and analyzing the one or more signals using one or more learning models to identify a request to perform the interpupillary measurement. In some examples of the methods, apparatuses, and non-transitory computer-readable media described herein, determining the conditions for performing the interpupillary measurement at the device may be based on identifying the request to perform the interpupillary measurement using the analysis of the one or more signals using the one or more learning models.

[0013] In some examples of the methods, apparatuses, and non-transitory computer-readable media described herein, the one or more signals include one or more audio signals associated with a user of the device, or one or more gestures associated with the user of the device, or both. In some examples of the methods, apparatuses, and non-transitory computer-readable media described herein, the one or more learning models include an audio recognition model or a gesture recognition model, or both.

[0014] Some examples of the methods, apparatuses, and non-transitory computer-readable media described herein may also include operations, features, units, or instructions for determining distortion correction parameters based on measurements of the set of interpupillary distances. In some examples of the methods, apparatuses, and non-transitory computer-readable media described herein, the calibration of the apparatus may be based on the distortion correction parameters.

[0015] In some examples of the methods, apparatuses and non-transitory computer-readable media described herein, the image set may further include a third subset of images captured by the first camera and a fourth subset of images captured by the second camera.

[0016] Some examples of the methods, apparatuses, and non-transitory computer-readable media described herein may also include operations, features, units, or instructions for: determining that the set of facial features is absent in each of the third and fourth image subsets; and avoiding remeasurement of the set of interpupillary distances based on the determination that the set of facial features is absent in each of the third and fourth image subsets.

[0017] Some examples of the methods, apparatuses, and non-transitory computer-readable media described herein may also include operations, features, units, or instructions for: determining that the set of facial features is absent in each of the third and fourth image subsets; and avoiding re-determining the interpupillary distance parameter based on the determination that the set of facial features is absent in each of the third and fourth image subsets.

[0018] Some examples of the methods, apparatuses, and non-transitory computer-readable media described herein may also include operations, features, units, or instructions for: determining that the set of facial features is absent in each of the third and fourth image subsets; and ignoring the third or fourth image subset, or both, based on the determination that the set of facial features is absent in each of the third and fourth image subsets.

[0019] In some examples of the methods, apparatuses, and non-transitory computer-readable media described herein, the set of facial features includes an iris set.

[0020] Some examples of the methods, apparatuses, or non-transitory computer-readable media described herein may also include operations, features, units, or instructions for determining stereo matching between a first iris in the iris set and a second iris in the iris set based on the detection of the set of facial features. In some examples of the methods, apparatuses, or non-transitory computer-readable media described herein, the stereo matching includes subpixel stereo matching.

[0021] Some examples of the methods, apparatuses, or non-transitory computer-readable media described herein may also include operations, features, units, or instructions for determining a stereo baseline associated with the iris set based on the detection of the facial feature set. In some examples of the methods, apparatuses, or non-transitory computer-readable media described herein, the determination of the stereo matching may be based on the stereo baseline.

[0022] In some examples of the methods, apparatuses, and non-transitory computer-readable media described herein, the camera set may each be located on an outward-facing surface of the device.

[0023] In some examples of the methods, apparatuses and non-transitory computer-readable media described herein, the camera set includes an eye-tracking camera set, a red-green-blue (RGB) camera set, an infrared (IR) camera set, or a time-of-flight (ToF) sensor set, or a combination thereof.

[0024] Some examples of the methods, apparatuses, and non-transitory computer-readable media described herein may also include operations, features, units, or instructions for calibrating the device based on one or more user profiles. Attached Figure Description

[0025] Figure 1 An example of a system supporting a head-mounted display device according to aspects of this disclosure is shown.

[0026] Figure 2 An example of a method for supporting a head-mounted display device according to aspects of this disclosure is shown.

[0027] Figure 3 and Figure 4 A block diagram of a device supporting a head-mounted display device according to aspects of this disclosure is shown.

[0028] Figure 5 A block diagram of a distortion correction manager supporting a head-mounted display device according to aspects of this disclosure is shown.

[0029] Figure 6A diagram of a system including a device supporting a head-mounted display device, according to aspects of this disclosure, is shown.

[0030] Figures 7 to 9 A flowchart illustrating a method for supporting a head-mounted display device according to aspects of this disclosure is shown. Detailed Implementation

[0031] Head-mounted displays have increasingly become an integral part of how users interact with various applications, such as augmented reality. These devices can be configured with display interfaces, cameras, and other hardware or software components to support a wide range of applications. Some head-mounted displays may experience lens distortion and can therefore be pre-configured to correct for it by assuming a fixed interpupillary distance (IPD) for the user. However, fixed IPD-based lens distortion correction can introduce visual artifacts to the user. Alternatively, some head-mounted displays can be configured to use an eye-tracking camera to calculate the IPD. However, the use of an eye-tracking camera can be resource-intensive and increase the power consumption and heat dissipation of the head-mounted display. Therefore, improvements for correcting IPD-based lens distortion may be desirable.

[0032] Various aspects of this disclosure relate to techniques for correcting lens distortion based on interpupillary distance using a stereo camera in a head-mounted display device (e.g., an augmented reality head-mounted display device). The described techniques can use one or more outward-facing (e.g., externally facing) cameras to determine the interpupillary distance of a user wearing the head-mounted display device. The head-mounted display device can be configured to trigger an interpupillary distance calculation state, or can be triggered via another device in electronic communication with the head-mounted display device, thereby switching the tracking camera to a streaming mode. The head-mounted display device can detect the user's eyes in tracking camera frames, determine the interpupillary distance using stereo images captured from the head-mounted display device, and refine the interpupillary distance estimate across multiple frames captured from multiple directions. After the interpupillary distance is determined, the head-mounted display device can determine optimal lens distortion correction parameters for the interpupillary distance to improve the visual quality of virtual objects rendered via the head-mounted display device.

[0033] First, various aspects of this disclosure are described in the context of the system. These aspects are further illustrated and described with reference to apparatus diagrams, system diagrams, and flowcharts relating to head-mounted display devices.

[0034] Figure 1An example of a system 100 supporting a head-mounted display device according to aspects of this disclosure is shown. System 100 may include device 105, server 110, and database 115. Although system 100 shows two devices 105, a single server 110, a single database 115, and a single network 120, this disclosure is applicable to any system architecture having one or more devices 105, server 110, database 115, and network 120. Device 105, server 110, and database 115 may communicate with each other and exchange information supporting the head-mounted display device, such as packets, data, or control information, via communication link 125 through network 120. In some cases, some or all of the distortion correction techniques described herein may be implemented by device 105 or server 110 or both.

[0035] Device 105 can be a head-mounted display or a handheld device (e.g., a smartphone with a camera). For example, device 105-a can be augmented reality glasses, a head-mounted display, etc. As a head-mounted display, device 105-a can be worn by user 155. In some examples, device 105-a can be configured with one or more sensors to sense the position of user 155 and / or the environment around the device to generate information when user 155 wears device 105-a. This information may include movement information, orientation information, angle information, etc., about device 105-a. In some cases, device 105-a can be configured with a microphone for capturing audio and one or more speakers for broadcasting audio. Device 105-a can also be configured with a set of lenses and a display screen for user 155 to view and become part of a virtual reality experience.

[0036] Device 105-a can be configured to perform lens distortion correction based on the interpupillary distance associated with user 155. The interpupillary distance of user 155 can vary between 50mm and 75mm. In some cases, device 105-a can be configured to support static interpupillary distance-based lens distortion correction. However, static interpupillary distance-based lens distortion correction may cause visual artifacts in user 155 wearing device 105-a. In some examples, if the interpupillary distance associated with user 155 is known (e.g., the user of the augmented reality glasses is known), device 105-a can be able to configure lens distortion correction parameters for that interpupillary distance. As a result, user 155 can experience improved visual quality of virtual objects rendered on device 105-a (e.g., augmented reality glasses).

[0037] Device 105-a can use a set of cameras to determine interpupillary distance. As a head-mounted display device, device 105-a may include an eye-facing side facing the eyes of user 155 when device 105-a is worn, and an outer side opposite the eye-facing side. Device 105-a may be configured to use an outer-facing set of cameras, such as cameras 130 and 135. Camera 130 or camera 135, or both, may be an eye-tracking camera (e.g., a 6-DOF head-tracking camera), an RGB camera set, an infrared (IR) camera set, or a time-of-flight (ToF) sensor set, or a combination thereof. Cameras 130 and 135 may be configured (e.g., operatively coupled to) the outer front surface of device 105-a. For example, cameras 130 and 135 may be part of the outer-facing body of device 105-a. In some examples, device 105-a may be a pair of augmented reality glasses and the set of cameras may be located on the outer side of the augmented reality glasses. Camera 130 may be located on the right side of device 105-a, and camera 135 may be located on the left side of device 105-a or on the outer front surface of device 105-a. Although device 105-a shows two cameras, this disclosure applies to any device architecture with two or more cameras.

[0038] Devices 105 can be configured to communicate wirelessly or directly (e.g., via a direct interface) with each other. For example, device 105-b (e.g., a smartphone) and device 105-a (e.g., a head-mounted display) can be able to communicate directly with each other (e.g., using peer-to-peer (P2P) or device-to-device (D2D) protocols). In some examples, device 105-a (e.g., augmented reality glasses) can determine the conditions for performing interpupillary measurement at device 105-a. Device 105-a (e.g., the head-mounted display) can enable the interpupillary measurement state of device 105-a based on conditions. For example, device 105-a (e.g., augmented reality glasses) can trigger the interpupillary measurement state, or device 105-b (e.g., paired with a smartphone) can trigger the interpupillary measurement state. In some examples, device 105-a can detect one or more signals based on one or more sensors of device 105-a, and use one or more learning models to analyze one or more signals to identify a request to perform an interpupillary measurement. The one or more signals may include one or more audio signals associated with user 155 of device 105-a, or one or more gestures associated with user 155 of device 105-a, or both. The one or more learning models may be audio recognition models or gesture recognition models, or both.

[0039] Device 105-a can enable cameras 130 and 135 and switch them to streaming mode. In streaming mode, user 155 can hold device 105-a (e.g., augmented reality glasses) in their hand and position device 105-a so that device 105-a faces user 155. When device 105-a (e.g., augmented reality glasses) is held at a distance (e.g., an arm's length) from user 155 and facing user 155, device 105-a can perform a calibration process. Device 105-a can perform the calibration process based on a distance that meets a threshold distance from user 155's face.

[0040] As part of the calibration process, device 105-a can use its camera set to capture a set of images in terms of orientation. For example, device 105-a can use its camera set to capture a set of images in terms of direction and angle. This set of images may include a first subset of images captured by camera 130 and a second subset of images captured by camera 135. In some examples, the images may be stereoscopic images. Device 105-a can then detect a set of facial features associated with user 155 in each of the first and second image subsets. For example, device 105-a can detect a set of irises (e.g., user 155's eyes) in the captured images (e.g., camera frames) from both camera 130 and camera 135.

[0041] Device 105-a can measure a set of interpupillary distances associated with the eyes of user 155 on this orientation set in each of the first and second image subsets. That is, device 105-a can calculate the interpupillary distance using stereo images captured from device 105-a. In some examples, device 105-a can be configured to use epipolar geometric constraints to determine the multidimensional position of each eye of user 155 and calculate the distance between the two eyes (e.g., interpupillary distance). Device 105-a can use multiple such measurements to improve accuracy. In some cases, intrinsic and extrinsic camera parameters of camera 130 or camera 135, or both, can be pre-configured for device 105-a.

[0042] Device 105-a can determine its interpupillary distance parameters based on aggregating the set of interpupillary distances over the set of orientations. For example, device 105-a can refine the interpupillary distance estimates over multiple frames acquired from multiple angles, as described herein. Once the interpupillary distances are calculated, device 105-a can forward them to its display pipeline to perform distortion correction for the geometry of device 105-a in relation to the facial geometry of user 155. Although the above operations are described with reference to a head-mounted display device (e.g., device 105-a), these operations can also be performed by device 105-b (e.g., a smartphone). That is, system 100 can support distortion correction using one or more cameras (e.g., smartphone cameras) of device 105-b, instead of using cameras from device 105-a.

[0043] Additionally or alternatively, device 105 may be referred to by those skilled in the art as user equipment (UE), user device, smartphone, Bluetooth device, Wi-Fi device, mobile station, subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, access terminal, mobile terminal, wireless terminal, remote terminal, handheld device, user agent, mobile client, client, and / or some other suitable term. In some cases, device 105 may also be able to communicate directly with another device (e.g., using peer-to-peer (P2P) or device-to-device (D2D) protocols). For example, device 105 may be able to receive or send various information, such as instructions or commands, to another device 105.

[0044] Device 105 may include distortion correction manager 150, which may support methods for performing one or more of the functions described herein. In some cases, device 105 may receive (e.g., download, stream, broadcast) data from server 110, database 115, or another device 105, or send (e.g., upload) data to server 110, database 115, or another device 105 via communication link 125. Distortion correction manager 150 may be a part of a general-purpose processor, digital signal processor (DSP), image signal processor (ISP), central processing unit (CPU), graphics processing unit (GPU), microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), discrete gate or transistor logic component, discrete hardware component, or any combination thereof, or other programmable logic device, discrete gate or transistor logic, discrete hardware component, or any combination thereof designed to perform the functions described herein, etc. For example, distortion correction manager 150 can process data (e.g., image data, video data, audio data) from local memory or database 115 of device 105 and / or write data to local memory or database 115 of device 105.

[0045] The distortion correction manager 150 can also be configured to provide enhancement, restoration, analysis, compression, streaming, and compositing functions, among others. For example, the distortion correction manager 150 can perform white balance, cropping, scaling (e.g., compression), resolution adjustment, stitching, color processing, filtering, spatial filtering, artifact removal, frame rate adjustment, encoding, decoding, and filtering. By further example, according to the techniques described herein, the distortion correction manager 150 can process data to support distortion correction for head-mounted display devices.

[0046] Server 110 can be a data server, cloud server, server associated with a subscription provider, proxy server, network server, application server, communication server, home server, mobile server, or any combination thereof. In some cases, server 110 may include distribution platform 140. Distribution platform 140 can allow device 105 to discover, browse, share, and download data via network 120 using communication link 125, and thus provide digital distribution of data from distribution platform 140. Therefore, digital distribution can be in the form of transmitting media content such as audio, video, and images without using physical media but via online transmission media (e.g., the Internet). For example, device 105 can upload or download applications for streaming, downloading, uploading, processing, and enhancing images, audio, video, etc. Server 110 can also send various information to device 105, such as instructions or commands for downloading applications on device 105.

[0047] Database 115 can store various types of information, such as instructions or commands. For example, database 115 can store content 145. Device 105 can retrieve the stored content 145 from database 115 via network 120 using communication link 125. In some examples, database 115 can be a relational database (e.g., a relational database management system (RDBMS) or a structured query language (SQL) database), a non-relational database, a network database, an object-oriented database, or other types of databases that store various types of information (e.g., instructions or commands).

[0048] Network 120 may provide encryption, access authorization, tracking, Internet Protocol (IP) connectivity, and other access, computation, modification, and / or functions. Examples of network 120 may include any combination of the following: cloud network, local area network (LAN), wide area network (WAN), virtual private network (VPN), wireless network (e.g., using 802.11), cellular network (using third-generation (3G), fourth-generation (4G), Long Term Evolution (LTE), or New Radio (NR) systems (e.g., fifth-generation (5G)), etc. Network 120 may include the Internet.

[0049] The communication link 125 shown in system 100 may include uplink transmissions from device 105 to server 110 and database 115, and / or downlink transmissions from server 110 and database 115 to device 105. Communication link 125 may enable bidirectional and / or unidirectional communication. In some examples, communication link 125 may be a wired connection, a wireless connection, or both. For example, communication link 125 may include one or more connections, including but not limited to Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), cellular, Z-WAVE, 802.11, point-to-point, LAN, wireless local area network (WLAN), Ethernet, FireWire, fiber optic, and / or other connection types associated with wireless communication systems.

[0050] The techniques described herein can provide improvements to head-mounted display devices. Furthermore, the techniques described herein can provide benefits and enhancements to the operation of device 105. For example, by providing accurate measurement of interpupillary distance without requiring an expensive eye-tracking camera on device 105, operational characteristics of device 105, such as power consumption, processor utilization (e.g., DSP, CPU, GPU, ISP processing utilization), and memory usage, can be reduced.

[0051] Figure 2 An example of a method 200 supporting a head-mounted display device according to aspects of this disclosure is shown. As described herein, operation of method 200 may be implemented by device 105 or its components. For example, operation of method 200 may be provided by reference to... Figure 1The described device is used to perform the functions. In some examples, device 105 may execute an instruction set to control the functional units of device 105 to perform the described functions. Additionally or alternatively, device 105 may use dedicated hardware to perform various aspects of the described functions.

[0052] Device 105 can capture a set of images (e.g., stereo frames) in an orientation set using its set of cameras (e.g., cameras 130 and 135). Cameras 130 and 135 can be outward-facing cameras. In this example, device 105 is a pair of augmented reality glasses, and cameras 130 and 135 can be located on the outer surface of the augmented reality glasses. That is, cameras 130 and 135 can be located on the outside of the augmented reality glasses, opposite to the eye-facing side of the glasses.

[0053] The image set may include a first image captured by camera 130 and a second image captured by camera 135. For example... Figure 2 As shown, device 105 can capture the set of images at a set of angles using its set of cameras (e.g., cameras 130 and 135). For example, a first image captured by camera 130 may be at a first angle, and a second image captured by camera 135 may also be at a first angle. Device 105 can then capture a third image using camera 130 and a fourth image using camera 135. The third image captured by camera 130 and the fourth image captured by camera 135 may be at a second angle different from the first angle.

[0054] One or more images (e.g., camera frames) may be forwarded as input to face detector 205. Face detector 205 may detect a set of facial features in the set of images. In some examples, face detector 205 may detect a set of facial features (e.g., iris features, and other facial features) in each of the first and second images. In some other examples, face detector 205 may determine that the set of facial features is not present in each of the third and fourth images. Face detector 205 may output key points around the user's eyes 235. Key points around the eyes may include the center of the left eye, the center of the right eye, the inner corner of the left eye, the outer corner of the left eye, the inner corner of the right eye, or the outer corner of the right eye, or any combination thereof. If no face is detected in two images (e.g., frames), the remainder of method 200 terminates (e.g., is skipped by device 105).

[0055] The output of face detector 205 can be forwarded as input to iris detector 210, which detects the iris in each image (e.g., frame). Stereo matching 215 matches, for example, the iris detected in the left image with the iris in the right image to obtain subpixel-accurate stereo matching. For example, device 105-a can determine a stereo match between a first iris and a second iris based on the detection of a set of facial features. Stereo matching can include subpixel stereo matching. In some examples, device 105-a can determine a stereo baseline based on the detection of a set of facial features. In some examples, device 105 can determine a stereo match based on a stereo baseline.

[0056] Interpupillary distance aggregation 220 can calculate the interpupillary distance based on the current stereo matching result and aggregate the information over several iterations (e.g., seconds) to produce an accurate interpupillary distance estimate. In some examples, interpupillary distance aggregation 220 can calculate the interpupillary distance based on information provided by camera calibration 225 (e.g., external and internal camera parameters). Interpupillary distance aggregation 220 can forward the interpupillary distance to the display pipeline 230 of device 105 to achieve distortion correction of the device 105 geometry in relation to the user's facial geometry.

[0057] Figure 3 A block diagram 300 illustrates a device 305 supporting a head-mounted display device according to aspects of this disclosure. Device 305 may be an example of an aspect of the head-mounted display device described herein. Device 305 may include a sensor 310, a display 315, and a distortion correction manager 320. Device 305 may also include a processor. Each of these components may communicate with each other (e.g., via one or more buses).

[0058] One or more sensors 310 (e.g., image sensors, cameras, etc.) can receive information (e.g., light, such as visible and / or invisible light), which can be transmitted to other components of device 305. In some cases, sensor 310 can be a reference. Figure 6 Examples of aspects of the described I / O controller 610. Sensor 310 may utilize one or more photosensitive elements sensitive to the electromagnetic radiation spectrum to receive information (e.g., sensor 310 may be configured or tuned to receive pixel intensity values, red-green-blue (RGB) values, infrared (IR) light values, near-IR light values, ultraviolet (UV) light values ​​of pixels, etc.). The information can then be passed to other components of device 305.

[0059] Display 315 can display content generated by other components of the device. Display 315 can be a reference. Figure 6An example of a display 635 is described. In some examples, the display 635 may be connected to a display buffer that stores rendered data until it is ready to display an image (e.g., as shown in the reference). Figure 6 (As described). The display 315 may emit light based on signals or information generated by other components of the device 305. For example, the display 315 may receive display information (e.g., pixel mapping, display adjustment) from the sensor 310 and may emit light accordingly. The display 315 may represent a unit capable of displaying video, images, text, or any other type of data for consumption by a viewer.

[0060] Display 315 may include a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic LED (OLED), an active-matrix OLED (AMOLED), etc. In some cases, display 315 and the I / O controller (e.g., I / O controller 610) may be or represent aspects of the same component (e.g., a touchscreen) of device 305. Display 315 may be any suitable display or screen that allows user interaction and / or allows the presentation of information (e.g., captured images and videos) for user viewing. In some aspects, display 315 may be a touch-sensitive display. In some cases, display 315 may display images captured by sensors, wherein the displayed image captured by the sensors may depend on the configuration of the light source and active sensors by the distortion correction manager 320.

[0061] The distortion correction manager 320, sensor 310, display 315, or various combinations thereof, or various components thereof, may be examples of units for performing various aspects of the head-mounted display device described herein. For example, the distortion correction manager 320, sensor 310, display 315, or various combinations thereof, or components thereof, may support methods for performing one or more of the functions described herein.

[0062] In some examples, the distortion correction manager 320, sensor 310, display 315, or various combinations or components thereof may be implemented in hardware (e.g., in communication management circuitry). The hardware may include any combination of the foregoing, such as a processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or a unit configured to or otherwise support the functions described herein. In some examples, the processor and memory coupled to the processor may be configured to perform one or more of the functions described herein (e.g., by executing instructions stored in memory by the processor).

[0063] Additionally or alternatively, in some examples, the distortion correction manager 320, sensor 310, display 315, or various combinations or components thereof may be implemented as processor-executable code (e.g., as communication management software or firmware). If implemented as processor-executable code, the functionality of the distortion correction manager 320, sensor 310, display 315, or various combinations or components thereof may be performed by a general-purpose processor, DSP, CPU, ASIC, FPGA, or any combination of these or other programmable logic devices (e.g., units configured or otherwise supported for performing the functions described in this disclosure).

[0064] In some examples, the distortion correction manager 320 can be configured to perform various operations (e.g., receive, monitor, transmit) using sensor 310, display 315, or both, or otherwise in conjunction with sensor 310, display 315, or both. For example, the distortion correction manager 320 can receive information from sensor 310, transmit information to display 315, or integrate with sensor 310, display 315, or both to receive information, transmit information, or perform various other operations described herein.

[0065] The distortion correction manager 320 may support distortion correction at the device 405 as disclosed herein. For example, the distortion correction manager 320 may be configured or otherwise support a unit for capturing an image set on an orientation set using a set of cameras of the device, the image set including a first subset of images captured by a first camera in the set of cameras and a second subset of images captured by a second camera in the set of cameras. The distortion correction manager 320 may be configured or otherwise support a unit for detecting a set of facial features in each of the first and second image subsets.

[0066] The distortion correction manager 320 can be configured or otherwise supported to include units for measuring a set of interpupillary distances on an orientation set based on a set of facial features in each of a first and second image subset. The distortion correction manager 320 can also be configured or otherwise supported to include units for determining interpupillary distance parameters for the device based on the aggregated set of interpupillary distances on the orientation set. The distortion correction manager 320 can also be configured or otherwise supported to include units for calibrating the device based on the interpupillary distance parameters.

[0067] By including or configuring a distortion correction manager 320 according to the examples described herein, device 305 (e.g., a processor that controls or otherwise couples to sensor 310, display 315, distortion correction manager 320, or a combination thereof) can support techniques for reducing processing, lowering power consumption, and utilizing device resources more efficiently.

[0068] Figure 4 A block diagram 400 illustrates a device 405 supporting a head-mounted display device according to aspects of this disclosure. Device 405 may be an example of aspects of device 305 or device 105 as described herein. Device 405 may include a sensor 410, a display 415, and a distortion correction manager 420. Device 405 may also include a processor. Each of these components may communicate with each other (e.g., via one or more buses).

[0069] One or more sensors 410 (e.g., image sensors, cameras, etc.) can receive information (e.g., light, such as visible and / or invisible light), which can be transmitted to other components of device 405. In some cases, sensor 410 can be a reference. Figure 6 Examples of aspects of the described I / O controller 610. Sensor 410 may utilize one or more photosensitive elements sensitive to the electromagnetic radiation spectrum to receive information (e.g., sensor 410 may be configured or tuned to receive pixel intensity values, red-green-blue (RGB) values, infrared (IR) light values, near-IR light values, ultraviolet (UV) light values ​​of pixels, etc.). The information can then be passed to other components of device 405.

[0070] Display 415 can display content generated by other components of the device. Display 415 can be a reference. Figure 6 An example of a display 635 is described. In some examples, the display 635 may be connected to a display buffer that stores rendered data until it is ready to display an image (e.g., as shown in the reference). Figure 6 (As described). The display 415 may emit light based on signals or information generated by other components of the device 405. For example, the display 415 may receive display information (e.g., pixel mapping, display adjustment) from the sensor 410 and may emit light accordingly.

[0071] Display 415 may represent a unit capable of displaying video, images, text, or any other type of data for a viewer to consume. Display 415 may include LCD, LED displays, OLED, AMOLED, etc. In some cases, display 415 and I / O controller (e.g., I / O controller 610) may be or represent aspects of the same component (e.g., touchscreen) of device 405. Display 415 may be any suitable display or screen that allows user interaction and / or allows the presentation of information (e.g., captured images and videos) for user viewing. In some aspects, display 415 may be a touch-sensitive display. In some cases, display 415 may display images captured by sensors, wherein the displayed image captured by the sensors may depend on the configuration of the light source and active sensors by the distortion correction manager 420.

[0072] Device 405 or its components may be examples of units described herein for performing various aspects of distortion correction. For example, distortion correction manager 420 may include camera component 425, recognition component 430, analysis component 435, calibration component 540, or any combination thereof. Distortion correction manager 420 may be an example of aspects of distortion correction manager 320 described herein. In some examples, distortion correction manager 420 or its components may be configured to: use sensor 410, display 415, or both, or otherwise cooperate with sensor 410, display 415, or both to perform various operations (e.g., receiving, monitoring, transmitting). For example, distortion correction manager 420 may receive information from sensor 410, transmit information to display 415, or integrate with sensor 410, display 415, or both to receive information, transmit information, or perform various other operations described herein.

[0073] Distortion correction manager 420 can support distortion correction at the device according to the present disclosure. Camera component 425 can be configured or otherwise supported for capturing a set of images on an orientation set using a set of cameras of the device, the image set including a first subset of images captured by a first camera in the set of cameras and a second subset of images captured by a second camera in the set of cameras. Recognition component 430 can be configured or otherwise supported for detecting a set of facial features in each of the first and second image subsets. Analysis component 435 can be configured or otherwise supported for measuring a set of interpupillary distances on the orientation set based on the set of facial features in each of the first and second image subsets. Analysis component 435 can be configured or otherwise supported for determining an interpupillary distance parameter for the device based on aggregating the interpupillary distance set on the orientation set. Calibration component 440 can be configured or otherwise supported for calibrating the device based on the interpupillary distance parameter.

[0074] Figure 5 A block diagram 500 illustrates a distortion correction manager 520 supporting a head-mounted display device according to aspects of this disclosure. The distortion correction manager 520 may be an example of the distortion correction manager 320, distortion correction manager 420, or aspects thereof described herein. The distortion correction manager 520 or its various components may be examples of units described herein for performing various aspects of distortion correction. For example, the distortion correction manager 520 may include a camera component 525, a recognition component 530, an analysis component 535, a calibration component 540, a triggering component 545, a status component 550, a sensor component 555, or any combination thereof. Each of these components may communicate directly or indirectly with each other (e.g., via one or more buses).

[0075] Distortion correction manager 520 can support distortion correction at the device according to the present disclosure. Camera component 525 can be configured or otherwise supported for capturing a set of images on an orientation set using a set of cameras of the device, the image set including a first subset of images captured by a first camera in the set of cameras and a second subset of images captured by a second camera in the set of cameras. Recognition component 530 can be configured or otherwise supported for detecting a set of facial features in each of the first and second image subsets. Analysis component 535 can be configured or otherwise supported for measuring a set of interpupillary distances on the orientation set based on the set of facial features in each of the first and second image subsets. In some examples, analysis component 535 can be configured or otherwise supported for determining an interpupillary distance parameter for the device based on aggregating the interpupillary distance set on the orientation set. Calibration component 540 can be configured or otherwise supported for calibrating the device based on the interpupillary distance parameter.

[0076] In some examples, trigger component 545 may be configured or otherwise supported to include units for determining the conditions for performing interpupillary measurement at the device. In some examples, state component 550 may be configured or otherwise supported to include units for conditionally enabling the interpupillary measurement state of the device, wherein the set of measured interpupillary distances is based on the enabling of the interpupillary measurement state. In some examples, trigger component 545 may be configured or otherwise supported to include units for receiving a request from the device, wherein determining the conditions for performing interpupillary measurement at the device is based on receiving a request from the device.

[0077] Sensor component 555 may be configured or otherwise support units for detecting one or more signals based on one or more sensors of the device. In some examples, sensor component 555 may be configured or otherwise support units for analyzing one or more signals using one or more learning models to identify a request to perform an interpupillary measurement. In some examples, trigger component 545 may be configured or otherwise support units for determining the conditions for performing an interpupillary measurement at the device based on identifying a request to perform an interpupillary measurement using one or more learning models. In some examples, the one or more signals include one or more audio signals (e.g., voice input) associated with a user of the device, or one or more gestures associated with a user of the device, or both. In some examples, the one or more learning models include an audio recognition model or a gesture recognition model, or both.

[0078] Analysis component 535 may be configured or otherwise supported to include units for determining distortion correction parameters based on a set of measured interpupillary distances, wherein device calibration is based on distortion correction parameters. In some examples, the image set may also include a third subset of images captured by a first camera and a fourth subset of images captured by a second camera. In some examples, analysis component 535 may be configured or otherwise supported to include units for determining that a subset of facial features is absent in each of the third and fourth image subsets. In some examples, analysis component 535 may be configured or otherwise supported to include units for avoiding remeasurement of the set of interpupillary distances based on the determination that a subset of facial features is absent in each of the third and fourth image subsets.

[0079] In some examples, analysis component 535 may be configured or otherwise supported to enable units for determining that a subset of facial features is absent in each of the third and fourth image subsets. In some examples, analysis component 535 may be configured or otherwise supported to avoid redetermining the interpupillary distance parameter based on the determination that a subset of facial features is absent in each of the third and fourth image subsets. In some examples, analysis component 535 may be configured or otherwise supported to enable units for determining that a subset of facial features is absent in each of the third and fourth image subsets. In some examples, analysis component 535 may be configured or otherwise supported to ignore either the third or fourth image subset, or both, based on the determination that a subset of facial features is absent in each of the third and fourth image subsets.

[0080] In some examples, the facial feature set includes an iris set. In some examples, the analysis component 535 may be configured or otherwise supported for determining a stereo match between a first iris in the iris set and a second iris in the iris set based on the detected facial feature set, wherein the stereo match includes subpixel stereo matching. In some examples, the analysis component 535 may be configured or otherwise supported for determining a stereo baseline associated with the iris set based on the detected facial feature set, wherein the determination of the stereo match is based on the stereo baseline. In some examples, the camera set is located on an outward-facing surface of the device. In some examples, the camera set includes an eye-tracking camera set, an RGB camera set, an IR camera set, or a ToF sensor set, or a combination thereof. In some examples, the calibration device is based on one or more user profiles.

[0081] Figure 6A diagram of a system 600 including device 605 supporting a head-mounted display device, according to aspects of this disclosure, is shown. Device 605 may be an example of or include components of device 305, device 405, or device 105 described herein. Device 605 may include components for bidirectional voice and data communication, including components for transmitting and receiving communications, such as a distortion correction manager 620, an I / O controller 610, a memory 615, and a processor 625. These components may communicate electronically or be otherwise coupled (e.g., operative ground, communicative ground, functional ground, electronic ground, electrical ground) via one or more buses (e.g., bus 640).

[0082] I / O controller 610 can manage the input and output signals of device 605. I / O controller 610 can also manage peripheral devices not integrated into device 605. In some cases, I / O controller 610 can represent a physical connection or port to an external peripheral device. In some cases, I / O controller 610 can use, for example... The operating system or other known operating system. In some other cases, the I / O controller 610 may represent or interact with a modem, keyboard, mouse, touchscreen, or similar device. In some cases, the I / O controller 610 may be implemented as part of a processor (e.g., processor 625). In some cases, a user may interact with device 605 via the I / O controller 610 or via hardware components controlled by the I / O controller 610.

[0083] Memory 615 may include RAM and ROM. Memory 615 may store computer-readable, computer-executable code 630, which includes instructions that, when executed by processor 625, cause device 605 to perform the various functions described herein. Code 630 may be stored in a non-transitory computer-readable medium such as system memory or other types of memory. In some cases, code 630 may not be directly executable by processor 625, but may cause a computer (e.g., when compiled and executed) to perform the functions described herein. In some cases, among others, memory 615 may contain a BIOS that controls basic hardware or software operations, such as interaction with peripheral components or devices.

[0084] Processor 625 may include intelligent hardware devices (e.g., general-purpose processors, DSPs, CPUs, microcontrollers, ASICs, FPGAs, programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any combination thereof). In some cases, processor 625 may be configured to use a memory controller to operate a memory array. In other cases, the memory controller may be integrated into processor 625. Processor 625 may be configured to execute computer-readable instructions stored in memory (e.g., memory 615) to cause device 605 to perform various functions (e.g., functions or tasks supporting a head-mounted display device). For example, device 605 or components of device 605 may include processor 625 and memory 615 coupled to processor 625, processor 625 and memory 615 being configured to perform the various functions described herein.

[0085] The distortion correction manager 620 can support distortion correction at the device according to the disclosure herein. For example, the distortion correction manager 620 can be configured or otherwise support units for capturing a set of images on an orientation set using a set of cameras of the device, the image set including a first subset of images captured by a first camera in the set of cameras and a second subset of images captured by a second camera in the set of cameras. The distortion correction manager 620 can be configured or otherwise support units for detecting a set of facial features in each of the first and second image subsets.

[0086] The distortion correction manager 620 can be configured or otherwise supported to include units for measuring a set of interpupillary distances on an orientation set based on a set of facial features in each of a first and second image subset. The distortion correction manager 620 can also be configured or otherwise supported to include units for determining interpupillary distance parameters for the device based on the aggregated set of interpupillary distances on the orientation set. The distortion correction manager 620 can also be configured or otherwise supported to include units for calibrating the device based on the interpupillary distance parameters.

[0087] By including or configuring a distortion correction manager 620 according to examples as described herein, device 605 can support techniques for reducing latency, improving the user experience associated with reduced processing, reducing power consumption, utilizing device resources more efficiently, and extending battery life, among others.

[0088] The distortion correction manager 620 or its sub-components may be implemented in hardware, processor-executable code (e.g., software or firmware), or any combination thereof. If implemented in processor-executable code, the functionality of the distortion correction manager 620 or its sub-components may be performed by a general-purpose processor, DSP, ASIC, field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic unit, discrete hardware component, or any combination thereof designed to perform the functions described in this disclosure. The distortion correction manager 620 or its sub-components may be physically located in various locations, including distributed such that some functionality is implemented by one or more physical components at different physical locations. In some examples, according to various aspects of this disclosure, the distortion correction manager 620 or its sub-components may be separate and distinct components. In some examples, according to various aspects of this disclosure, the distortion correction manager 620 or its sub-components may be combined with one or more other hardware components, including but not limited to I / O components, camera controllers, another computing device, one or more other components described in this disclosure, or combinations thereof.

[0089] Figure 7 A flowchart illustrating a method 700 supporting a head-mounted display device according to aspects of this disclosure is shown. As described herein, operation of method 700 can be implemented by a device or its components. For example, operation of method 700 can be performed by reference to... Figures 1 to 6 The described device is used to perform this function. In some examples, the device may execute a set of instructions to control the functional units of the device to perform the described function. Additionally or alternatively, the device may use dedicated hardware to perform various aspects of the described function.

[0090] At 705, the method may include: capturing an image set on an orientation set using a set of cameras of the device, the image set including a first subset of images captured by a first camera in the set of cameras and a second subset of images captured by a second camera in the set of cameras. The operation at 705 can be performed according to the examples disclosed herein. In some examples, some aspects of the operation at 705 may be derived from, as referenced... Figure 5 The camera component 525 described herein is used to perform this action.

[0091] At 710, the method may include: detecting a set of facial features in each of the first and second image subsets. The operation at 710 can be performed according to the examples disclosed herein. In some examples, some aspects of the operation at 710 may be derived from, as referenced... Figure 5 The described recognition component 530 is used to perform this task.

[0092] At 715, the method may include: measuring a set of interpupillary distances on an orientation set based on a set of facial features in each of the first and second image subsets. The operation at 715 can be performed according to the examples disclosed herein. In some examples, aspects of the operation at 715 may be derived from, as referenced... Figure 5 The described analysis component 535 is used to perform this.

[0093] At 720, the method may include: determining interpupillary distance parameters for the device based on aggregating a set of interpupillary distances over an orientation set. The operation at 720 can be performed according to the examples disclosed herein. In some examples, aspects of the operation at 720 may be derived from, as referenced... Figure 5 The described analysis component 535 is used to perform this.

[0094] At 725, the method may include calibrating the device based on an interpupillary distance parameter. The operation at 725 can be performed according to the examples disclosed herein. In some examples, aspects of the operation at 725 may be derived from, as referenced... Figure 5 The calibration component 540 described herein is used to perform the calibration.

[0095] Figure 8 A flowchart illustrating a method 800 supporting a head-mounted display device according to aspects of this disclosure is shown. As described herein, operation of method 800 can be implemented by a device or its components. For example, operation of method 800 can be performed by reference to... Figures 1 to 6 The described device is used to perform this function. In some examples, the device may execute a set of instructions to control the functional units of the device to perform the described function. Additionally or alternatively, the device may use dedicated hardware to perform various aspects of the described function.

[0096] At 805, the method may include: determining the conditions for performing interpupillary measurement. The operation at 805 can be performed according to the examples disclosed herein. In some examples, aspects of the operation at 805 may be derived from, as referenced... Figure 5 The described triggering component 545 is executed.

[0097] At 810, the method may include: enabling interpupillary measurement state based on the condition. The operation at 810 can be performed according to the examples disclosed herein. In some examples, aspects of the operation at 810 may be derived from, as referenced... Figure 5 The described state component 550 is used to execute.

[0098] At 815, the method may include: capturing an image set on an orientation set using a set of cameras of the device, the image set including a first subset of images captured by a first camera in the set of cameras and a second subset of images captured by a second camera in the set of cameras. The operation of 815 can be performed according to the examples disclosed herein. In some examples, some aspects of the operation of 815 may be derived from, as referenced... Figure 5 The camera component 525 described herein is used to perform this action.

[0099] At 820, the method may include: detecting a set of facial features in each of a first subset of images and a second subset of images. The operation at 820 can be performed according to the examples disclosed herein. In some examples, some aspects of the operation at 820 may be derived from, as referenced... Figure 5 The described recognition component 530 is used to perform this task.

[0100] At 825, the method may include: measuring a set of interpupillary distances on an orientation set based on a set of facial features in each of a first and second image subset. The operation at 825 can be performed according to the examples disclosed herein. In some examples, aspects of the operation at 825 may be derived from, as referenced... Figure 5 The described analysis component 535 is used to perform this.

[0101] At 830, the method may include: determining interpupillary distance parameters for the device based on aggregating a set of interpupillary distances over an orientation set. The operation of 830 can be performed according to the examples disclosed herein. In some examples, aspects of the operation of 830 may be derived from, as referenced... Figure 5 The described analysis component 535 is used to perform this.

[0102] At 835, the method may include calibrating the device based on an interpupillary distance parameter. Operation 835 can be performed according to the examples disclosed herein. In some examples, aspects of operation 835 may be derived from, as referenced... Figure 5 The calibration component 540 described herein is used to perform the calibration.

[0103] Figure 9 A flowchart illustrating a method 900 supporting a head-mounted display device according to aspects of this disclosure is shown. As described herein, operation of method 900 can be implemented by a device or its components. For example, operation of method 900 can be performed by reference to... Figures 1 to 6 The described device is used to perform this function. In some examples, the device may execute a set of instructions to control the functional units of the device to perform the described function. Additionally or alternatively, the device may use dedicated hardware to perform various aspects of the described function.

[0104] At 905, the method may include: capturing an image set on an orientation set using a set of cameras of the device, the image set including a first subset of images captured by a first camera in the set of cameras and a second subset of images captured by a second camera in the set of cameras. The operation at 905 can be performed according to the examples disclosed herein. In some examples, some aspects of the operation at 905 may be derived from, as referenced... Figure 5 The camera component 525 described herein is used to perform this action.

[0105] At 910, the method may include: detecting a set of facial features in each of the first and second image subsets. The operation at 910 can be performed according to the examples disclosed herein. In some examples, some aspects of the operation at 910 may be derived from, as referenced... Figure 5 The described recognition component 530 is used to perform this task.

[0106] At 915, the method may include: determining a stereo matching between a first iris in a set of irises and a second iris in a set of irises based on a detected set of facial features. Stereo matching may include subpixel stereo matching. The operation at 915 can be performed according to the examples disclosed herein. In some examples, aspects of the operation at 915 may be derived from, as referenced... Figure 5 The described analysis component 535 is used to perform this.

[0107] At 920, the method may include: measuring a set of interpupillary distances on an orientation set based on a set of facial features in each of a first and second image subset. The operation at 920 can be performed according to the examples disclosed herein. In some examples, aspects of the operation at 920 may be derived from, as referenced... Figure 5 The described analysis component 535 is used to perform this.

[0108] At 925, the method may include: determining interpupillary distance parameters for the device based on aggregating a set of interpupillary distances over an orientation set. The operation at 925 can be performed according to the examples disclosed herein. In some examples, aspects of the operation at 925 may be derived from, as referenced... Figure 5 The described analysis component 535 is used to perform this.

[0109] At 930, the method may include calibrating the device based on an interpupillary distance parameter. The operation of 930 can be performed according to the examples disclosed herein. In some examples, aspects of the operation of 930 may be derived from, as referenced... Figure 5 The calibration component 540 described herein is used to perform the calibration.

[0110] It should be noted that the methods described in this paper describe possible implementations, and the operations and steps can be rearranged or otherwise modified, and other implementations are possible. Furthermore, aspects from two or more of these methods can be combined.

[0111] The following provides an overview of aspects of this disclosure:

[0112] Aspect 1: A method for distortion correction at a device, comprising: capturing a set of images on an orientation set using a set of cameras of the device, the set of images including a first subset of images captured by a first camera in the set of cameras and a second subset of images captured by a second camera in the set of cameras; detecting a set of facial features in each of the first subset of images and the second subset of images; measuring a set of interpupillary distances on the orientation set based at least in part on the set of facial features in each of the first subset of images and the second subset of images; determining an interpupillary distance parameter for the device based at least in part on aggregating the set of interpupillary distances on the orientation set; and calibrating the device based at least in part on the interpupillary distance parameter.

[0113] Aspect 2: The method according to aspect 1 further includes: determining conditions for performing interpupillary measurement at the device; and enabling an interpupillary measurement state of the device at least in part based on the conditions, wherein the measurement of the set of interpupillary distances is at least in part based on the enabling of the interpupillary measurement state.

[0114] Aspect 3: The method according to aspect 2 further includes: receiving a request from the device, wherein determining the conditions for performing the interpupillary measurement at the device is at least partially based on receiving the request from the device.

[0115] Aspect 4: The method according to aspect 3 further includes: detecting one or more signals at least in part based on one or more sensors of the device; and using one or more learning models to analyze the one or more signals to identify a request to perform the interpupillary measurement, wherein determining the conditions for performing the interpupillary measurement at the device is based at least in part on identifying the request to perform the interpupillary measurement using the analysis of the one or more signals using the one or more learning models.

[0116] Aspect 5: According to the method of aspect 4, wherein the one or more signals include one or more audio signals associated with a user of the device, or one or more gestures associated with the user of the device, or both; and the one or more learning models include an audio recognition model or a gesture recognition model, or both.

[0117] Aspect 6: The method according to any one of aspects 1 to 5 further includes: determining distortion correction parameters based at least in part on measuring the set of interpupillary distances, wherein the calibration of the device is based at least in part on the distortion correction parameters.

[0118] Aspect 7: The method according to any of Aspects 1 to 6, wherein the image set further includes a third subset of images captured by the first camera and a fourth subset of images captured by the second camera.

[0119] Aspect 8: The method according to aspect 7 further includes: determining that the set of facial features does not exist in each of the third image subset and the fourth image subset; and avoiding re-measuring the set of interpupillary distances based at least in part on the determination that the set of facial features does not exist in each of the third image subset and the fourth image subset.

[0120] Aspect 9: The method according to any of Aspects 7 to 8 further includes: determining that the facial feature set is absent in each of the third image subset and the fourth image subset; and avoiding redetermining the interpupillary distance parameter based at least in part on the determination that the facial feature set is absent in each of the third image subset and the fourth image subset.

[0121] Aspect 10: The method according to any one of Aspects 7 to 9 further includes: determining that the set of facial features does not exist in each of the third image subset and the fourth image subset; and ignoring the third image subset or the fourth image subset, or both, at least in part based on the determination that the set of facial features does not exist in each of the third image subset and the fourth image subset.

[0122] Aspect 11: The method according to any one of aspects 1 to 10, wherein the set of facial features includes the set of irises.

[0123] Aspect 12: The method according to aspect 11 further includes: determining, at least in part, a stereo matching between a first iris in the iris set and a second iris in the iris set based on the detected set of facial features, wherein the stereo matching includes subpixel stereo matching.

[0124] Aspect 13: The method according to aspect 12 further includes: determining a stereo baseline associated with the iris set based at least in part on detecting the set of facial features, wherein the determination of the stereo matching is based at least in part on the stereo baseline.

[0125] Aspect 14: The method according to any one of aspects 1 to 13, wherein each of the camera sets is located on an outward-facing surface of the device.

[0126] Aspect 15: The method according to any of Aspects 1 to 14, wherein the camera set includes an eye-tracking camera set, an RGB camera set, an IR camera set, or a ToF sensor set, or a combination thereof.

[0127] Aspect 16: The method according to any one of Aspects 1 to 15, wherein the calibration of the device is based at least in part on one or more user profiles.

[0128] Aspect 17: An apparatus for distortion correction, comprising a processor, a memory coupled to the processor, and instructions stored in the memory and executable by the processor to cause the apparatus to perform the methods of any of Aspects 1 to 16.

[0129] Aspect 18: An apparatus for distortion correction, comprising at least one unit for performing the method described in any of aspects 1 to 16.

[0130] Aspect 19: A non-transitory computer-readable medium storing code for distortion correction at a device, the code including instructions executable by a processor to perform the methods described in any of Aspects 1 to 16.

[0131] The information and signals described herein can be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips mentioned throughout this specification can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or light particles, or any combination thereof.

[0132] Using a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any combination thereof designed to perform the functions described herein, the various illustrative blocks and modules described in connection with the disclosure herein may be implemented or executed. The general-purpose processor may be a microprocessor; however, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such architecture).

[0133] The functions described herein can be implemented using hardware, software executed by a processor, firmware, or any combination thereof. If implemented by software executed by a processor, these functions can be stored as one or more instructions or code on or transmitted over a computer-readable medium. Other examples and implementations are within the scope of this application and the appended claims. For example, due to the nature of software, the functions described herein can be implemented using software executed by a processor, hardware, firmware, hardwiring, or any combination thereof. Features implementing the functions can also be physically placed in various locations, including portions distributed such that functions are implemented at different physical locations.

[0134] Computer-readable media includes both non-transitory computer storage media and communication media, wherein the communication media includes any medium that facilitates the transfer of a computer program from one location to another. Non-transitory storage media can be any available medium that can be accessed by a general-purpose computer or a special-purpose computer. By way of example, and not limitation, non-transitory computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), flash memory, disc-on-CD (CD) ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer or a general-purpose or special-purpose processor. Furthermore, any connection can be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disk and disc include CDs, laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically. The above combinations should also be included within the scope of computer-readable media.

[0135] As used herein, the word "or" as used in the claims, as in the list of entries (e.g., a list of entries preceded by phrases such as "at least one of" or "one or more of"), indicates an inclusive list such that, for example, a list of at least one of A, B, or C means A, or B, or C, or AB, or AC, or BC, or ABC (i.e., A and B and C). Furthermore, as used herein, the phrase "based on" should not be construed as a reference to a closed set of conditions. For example, an exemplary step described as "based on condition A" may be based on both condition A and condition B without departing from the scope of this disclosure. In other words, as used herein, the phrase "based on" will be interpreted in the same manner as the phrase "at least partially based on".

[0136] In the accompanying drawings, similar components or features may have the same reference numerals. Additionally, components of the same type may be distinguished by a dash followed by a second reference numeral to differentiate between similar components. If only the first reference numeral is used in this specification, the description applies to any similar component having the same first reference numeral, regardless of the second reference numeral or other subsequent reference numerals.

[0137] The specification described herein, in conjunction with the accompanying drawings, describes exemplary configurations and does not represent all examples that can be implemented or that fall within the scope of the claims. The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration," and not "preferred" or "advantageous" relative to other examples. Specific details are included to provide an understanding of the described techniques. However, these techniques can be implemented without using these specific details. In some cases, well-known structures and devices are shown in block diagram form to avoid obscuring the concepts of the described examples.

[0138] The description herein is provided to enable those skilled in the art to implement or use the disclosed content. Various modifications to this disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is consistent with the broadest scope of the principles and novel features disclosed herein.

Claims

1. A method for distortion correction at a device, the device comprising a pair of augmented reality glasses and a set of cameras, the set of cameras being located outwardly of the pair of augmented reality glasses, the method comprising: capturing a set of images on a set of orientations using the set of cameras of the device, the set of images comprising a first subset of images captured by a first camera of the set of cameras and a second subset of images captured by a second camera of the set of cameras at a first orientation, wherein the set of images further comprises a third subset of images captured by the first camera and a fourth subset of images captured by the second camera at a second orientation; detecting a set of facial features in each of the first subset of images and the second subset of images; measuring a set of inter-pupillary distances on the set of orientations based at least in part on the set of facial features in each of the first subset of images and the second subset of images; determining an inter-pupillary distance parameter for the device based at least in part on aggregating the set of inter-pupillary distances on the set of orientations; and calibrating the device based at least in part on the inter-pupillary distance parameter.

2. The method of claim 1, further comprising: determining a condition for performing an inter-pupillary measurement at the device; and enabling an inter-pupillary measurement state of the device based at least in part on the condition, wherein measuring the set of inter-pupillary distances is based at least in part on the enabling the inter-pupillary measurement state.

3. The method of claim 2, further comprising: receiving a request from the device, wherein determining the condition for performing the inter-pupillary measurement at the device is based at least in part on the receiving the request from the device.

4. The method of claim 3, further comprising: detecting one or more signals based at least in part on one or more sensors of the device; and analyzing the one or more signals using one or more learning models to identify the request to perform the inter-pupillary measurement, wherein determining the condition for performing the inter-pupillary measurement at the device is based at least in part on the analyzing the one or more signals using the one or more learning models to identify the request to perform the inter-pupillary measurement.

5. The method of claim 4, wherein, the one or more signals comprise one or more audio signals associated with a user of the device, or one or more gestures associated with the user of the device, or both; and the one or more learning models comprise an audio recognition model or a gesture recognition model, or both.

6. The method of claim 1, further comprising: determining a distortion correction parameter based at least in part on measuring the set of inter-pupillary distances, wherein the calibrating the device is based at least in part on the distortion correction parameter.

7. The method of claim 1, further comprising: determining an absence of the set of facial features in each of the third subset of images and the fourth subset of images; and avoiding remeasuring the set of interpupillary distances based at least in part on determining that the set of facial features is not present in each of the third subset of images and the fourth subset of images.

8. The method of claim 1, further comprising: determining that the set of facial features is not present in each of the third subset of images and the fourth subset of images; and avoiding redetermining the interpupillary distance parameter based at least in part on determining that the set of facial features is not present in each of the third subset of images and the fourth subset of images.

9. The method of claim 1, further comprising: determining that the set of facial features is not present in each of the third subset of images and the fourth subset of images; and ignoring the third subset of images or the fourth subset of images, or both, based at least in part on determining that the set of facial features is not present in each of the third subset of images and the fourth subset of images.

10. The method of claim 1, wherein, The set of facial features includes a set of irises.

11. The method of claim 10, further comprising: determining a stereo match between a first iris of the set of irises and a second iris of the set of irises based at least in part on the detecting the set of facial features, wherein the stereo match includes a sub-pixel stereo match.

12. The method of claim 11, further comprising: determining a stereo baseline associated with the set of irises based at least in part on the detecting the set of facial features, wherein the determining the stereo match is based at least in part on the stereo baseline.

13. The method of claim 1, wherein, The set of cameras includes a set of eye tracking cameras, a set of red, green, blue cameras, a set of infrared cameras, or a set of time-of-flight sensors, or a combination thereof.

14. The method of claim 1, wherein, Calibrating the device is based at least in part on one or more user profiles.

15. An apparatus for distortion correction, comprising: a processor; a memory coupled with the processor; a pair of augmented reality glasses and a set of cameras, wherein the set of cameras are located outwardly of the pair of augmented reality glasses; and instructions stored in the memory and executable by the processor to cause the apparatus to: capture a set of images using the set of cameras of the apparatus over a set of orientations, the set of images including a first subset of images captured by a first camera of the set of cameras and a second subset of images captured by a second camera of the set of cameras at a first orientation, wherein the set of images further includes a third subset of images captured by the first camera and a fourth subset of images captured by the second camera at a second orientation; detect a set of facial features in each of the first subset of images and the second subset of images; measure a set of interpupillary distances over the set of orientations based at least in part on the set of facial features in each of the first subset of images and the second subset of images; determine an interpupillary distance parameter for the apparatus based at least in part on aggregating the set of interpupillary distances over the set of orientations; and calibrate the apparatus based at least in part on the interpupillary distance parameter.

16. The apparatus of claim 15, wherein, The instructions are further executable by the processor to cause the apparatus to: detecting one or more signals based at least in part on one or more sensors of the device; analyzing the one or more signals using one or more learning models to identify a request to perform the interpupillary measurement, and determining a condition to perform the interpupillary measurement at the device based at least in part on analyzing the one or more signals using the one or more learning models to identify the request to perform the interpupillary measurement.

17. An apparatus for distortion correction comprising a pair of augmented reality glasses and a set of cameras positioned outwardly of the pair of augmented reality glasses, the apparatus further comprising: means for capturing a set of images using the set of cameras of the apparatus over a set of orientations, the set of images comprising a first subset of images captured by a first camera of the set of cameras and a second subset of images captured by a second camera of the set of cameras at a first orientation, wherein the set of images further comprises a third subset of images captured by the first camera and a fourth subset of images captured by the second camera at a second orientation; means for detecting a set of facial features in each of the first subset of images and the second subset of images; means for measuring a set of interpupillary distances over the set of orientations based at least in part on the set of facial features in each of the first subset of images and the second subset of images; means for determining an interpupillary distance parameter for the apparatus based at least in part on aggregating the set of interpupillary distances over the set of orientations; and means for calibrating the apparatus based at least in part on the interpupillary distance parameter.

Images