Pairing with a companion device

Abstract

This disclosure relates to pairing with a companion device. Systems and methods for device authentication or pairing are disclosed herein. In one aspect, a wearable display system includes a display, an image capture device configured to capture an image of the companion device, a computer-readable storage medium configured to store the image of the companion device, and a processor in communication with the image capture device and the storage medium. The processor can be programmed with executable instructions to receive a first image of a first optical pattern displayed by the companion device captured by the image capture device, wherein the companion device generates the first optical pattern based on first shared data, extract first data from the first optical pattern in the received first image, authenticate the companion device based on the first data extracted from the first optical pattern, and notify a user of the wearable display system that the companion device is authenticated.

Description

[0001] This application is a divisional application of application number 201880033535.5, filed on May 17, 2018, entitled "Pairing with a Companion Device".

[0002] Cross-references to related applications

[0003] This application claims the benefit of priority to U.S. Patent Application 62 / 509,427, filed May 22, 2017, entitled “Pairing with a Companion Device,” the entire contents of which are incorporated herein by reference. Technical Field

[0004] This disclosure relates to systems and methods for secure data transmission, and more particularly to systems and methods for device pairing and authentication for secure data transmission. Background Technology

[0005] To securely transfer data between a user device (e.g., a head-mounted display) and a companion device (e.g., another user device, such as a mobile device or totem), the user device and companion device need to be paired. The pairing process may include the user device authenticating that the companion device is authorized to communicate with it. After authentication, the user device and companion device can establish shared information or data (e.g., a shared key, a shared secret, or a shared signing key), which can be used to encrypt data transmitted between the two devices. A typical pairing process between devices can be cumbersome and involves multiple steps, such as entering a personal identification number (PIN) on both devices. Summary of the Invention

[0006] In one aspect, a wearable display system is disclosed. The wearable display system includes: an image capturing device configured to capture an image of a companion device; a non-transitory computer-readable storage medium configured to store the image of the companion device and executable instructions; and a processor communicating with the image capturing device and the non-transitory computer-readable storage medium, the processor being programmed by the executable instructions to: receive a first image of a first optical pattern displayed by the companion device, captured by the image capturing device, wherein the first optical pattern is generated by the companion device based on first shared data; extract first data from the first optical pattern of the received first image; and authenticate the companion device based on the first data extracted from the first optical pattern.

[0007] In another aspect, a method for device authentication is disclosed. Under the control of a hardware processor, the method includes: receiving a data object via a communication channel; receiving an image of an optical pattern displayed by a companion device, wherein the companion device generates the optical pattern using shared data; extracting first data from the optical pattern in the received image; generating a transformation of the data object using the first data; and sending the transformation of the data object to the companion device via the communication channel for authentication.

[0008] In another aspect, a head-mounted display system is disclosed. This head-mounted display system includes: a non-transitory computer-readable storage medium storing executable instructions; and a processor communicating with the non-transitory computer-readable storage medium, the processor being programmed by the executable instructions to: receive first communication from a device, wherein the first communication is generated by the device based on first shared data; extract first data from the first communication; and authenticate the device based on the first data extracted from the first communication.

[0009] In another aspect, a head-mounted display system is disclosed. The head-mounted display system includes: a non-transitory computer-readable storage medium storing executable instructions; and a processor communicating with the non-transitory computer-readable storage medium, the processor being programmed by the executable instructions to: receive from a first device a first communication including a first data object, wherein the first communication is generated at least partially based on first shared data, wherein the first device is configured to send the first shared data to a third device; receive from a second device a second communication including a second data object, wherein the second communication is generated at least partially based on second shared data, wherein the second device is configured to send the second shared data to the third device; send the first data object and the second data object to the third device, wherein the third device is configured to authenticate the head-mounted display system, the first device, and the second device at least partially based on the first shared data from the first device, the second shared data from the second device, the first data object from the head-mounted display system, and the second data object from the head-mounted display system; and receive from the third device a third communication including a third data object, wherein the third data object indicates that the third device has successfully authenticated the head-mounted display system, the first device, and the second device.

[0010] This document discloses systems and methods for device authentication. In some embodiments, after receiving an image of an optical pattern displayed by a device based on shared information, the system or method can extract information from the optical pattern in the received image. The method can authenticate the device based on the information extracted from the optical pattern.

[0011] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the specification, drawings, and claims. This summary and the following detailed description are not intended to define or limit the scope of the subject matter of this disclosure. Attached Figure Description

[0012] Figure 1 Device authentication according to one embodiment is illustrated.

[0013] Figure 2 This is an interactive diagram illustrating device authentication using shared information or data from a user device according to one embodiment.

[0014] Figure 3 This is an interactive diagram illustrating device authentication using challenge text according to one embodiment.

[0015] Figure 4 This is an interactive diagram illustrating device authentication using shared information or data generated by a companion device according to one embodiment.

[0016] Figure 5 A flowchart of a device authentication method according to one embodiment is shown.

[0017] Figure 6 Device authentication according to one embodiment is illustrated.

[0018] Figure 7 It is an interaction diagram showing the device authentication between a user device and the two companion devices using visual patterns generated by two companion devices.

[0019] Figure 8 It is an interaction diagram showing device authentication between a user device and two companion devices using visual patterns generated at least partially by one or more devices in the cloud.

[0020] Figure 9 An illustration depicts an augmented reality scene according to one embodiment, featuring a specific virtual reality object and a specific real-world object viewed by a person.

[0021] Figure 10 An example of a wearable display system according to one embodiment is shown.

[0022] Figure 11 Various aspects of a method for simulating a three-dimensional image using multiple depth planes, according to one embodiment, are illustrated.

[0023] Figure 12An example of a waveguide stack for outputting image information or data to a user, according to one embodiment, is shown.

[0024] Figure 13 An example outgoing beam that can be output by a waveguide according to one embodiment is shown.

[0025] Figure 14 This is a schematic diagram illustrating a display system according to one embodiment.

[0026] Throughout the accompanying drawings, reference numerals may be used repeatedly to indicate correspondences between reference elements. These drawings are provided to illustrate the exemplary embodiments described herein and are not intended to limit the scope of this disclosure. Detailed Implementation

[0027] Overview

[0028] Wearable display systems, such as head-mounted displays (HMDs), can be configured to work with companion devices, such as handheld devices or user input devices (e.g., totems). In order for the HMD and companion devices to work together (e.g., wirelessly exchange information or data), the companion devices and the HMD may first need to be paired with each other. This disclosure provides systems and methods for pairing such devices.

[0029] The HMD may include an outward-facing image capture device, such as a camera, capable of imaging the environment of the HMD's user. The image capture device can be used to capture and acquire information or data (e.g., keys or secrets) displayed on a companion device (e.g., a totem with a display or light source) to establish shared information or data (e.g., shared keys, shared secrets, or signing keys) between the HMD and the companion device. The shared data is then used by the HMD or the companion device to encrypt / decrypt data used for / from secure data transmission. Systems and methods for efficient and secure device pairing and authentication are disclosed.

[0030] Figure 1 Device authentication according to one embodiment is illustrated. User device 104 can authenticate companion devices, such as totem 108a (e.g., used as a virtual user input device) or mobile phone 108b. With totem 108a, a user can interact with the input surface (e.g., touchpad) of totem 108a using their thumb and can hold totem 108a using other fingers. User device 104 can be an HMD, wearable display system, tablet, mobile phone, or mobile device. Companion devices (e.g., totem 108a and / or mobile phone 108b) can be HMDs, wearable display systems, tablets, totems, or mobile devices. References below... Figure 9-14 Examples describing HMDs and totems.

[0031] User equipment 104 and totem 108a can communicate wirelessly via radio frequency (RF) communication channels (such as Wi-Fi, Bluetooth, etc.). Communication within the bandwidth of an RF channel is typically referred to as in-band communication and can represent the primary communication channel between the two devices. User equipment 104 and totem 108a can also be configured to communicate via a second, different communication channel (e.g., optical channel, acoustic channel, near field communication (NFC) channel, etc.). This type of communication is typically referred to as out-of-band (OOB) communication.

[0032] Reference Figure 1 User equipment 104 can receive out-of-band communication 112 from companion devices (e.g., totem 108a and / or mobile phone 108b). Out-of-band communication 112 may include optical communication (e.g., one-dimensional barcodes, or two-dimensional codes, such as quick-response (QR) codes), audio communication (e.g., inaudible to the human ear or ultrasound), spoken phrases (e.g., alphanumeric phrases), one or more spectral characteristics of the spoken phrases, user biometric information or data (e.g., iris code), or any combination thereof. Optical communication may include visual communication (within the wavelength range perceptible to a human user, e.g., from about 400 nm to 700 nm) or non-visual communication (e.g., in the infrared or ultraviolet wavelength range). User equipment 104 can receive out-of-band communication 112 through explicit or implicit user interaction. For example, user equipment 104 may be in pairing mode and automatically receive out-of-band communication 112. As another example, user equipment 104 may use its camera to continuously monitor its surroundings to determine the presence of visual communication on the companion device.

[0033] Companion devices (e.g., totem 108a and / or mobile phone 108b) can generate and display visual patterns using their displays or other optical devices. For example, totem 108a may have a green light-emitting diode (LED) 116a and a red LED 116b. Totem 108a can use two LEDs to display a visual pattern. For example, to display the number "57", totem 108a may blink the green LED 5 times and the red LED 7 times. User device 104 can capture the pattern using its image capture device and extract the number "57" from the captured pattern using computer vision technology. Totem 108a can generate the visual pattern, and user device 104 can extract it from the visual pattern based on a predetermined protocol. For example, the predetermined protocol may specify that the shared data is the sum of ten times the number of blinks of the red LED 116b and the number of blinks of the green LED 116a. Advantageously, totem 108a does not need to include an image capture device or microphone for user device 104 to authenticate companion device 108.

[0034] As another example, mobile phone 108b can display the number "57" on its display 120. The number "57" can be displayed as part of a QR code (e.g., a two-dimensional pattern). User device 104 can capture the displayed number, QR code, or two-dimensional pattern using its image capture device. User device 104 can extract the number "57" from the captured image using computer vision technology. Mobile phone 108b can generate a visual pattern, and user device 104 can extract it from the visual pattern based on a predetermined protocol. Advantageously, device authentication does not require user input or initiation.

[0035] Device verification example using shared data from user devices

[0036] In some embodiments, shared information or data (e.g., shared secret, shared key, or signing key) may be sent by user equipment 104 and received by companion equipment 108. Figure 2 This is an interaction diagram illustrating device authentication using shared data from a user device according to one embodiment. User device 104 may generate the shared data at interaction 204. For example, user device 104 may generate a value randomly. As another example, the shared data may be predetermined. Subsequently, user device 104 may send the shared data to companion device 108 at interaction 208. For example, user device 104 may send a byte stream containing the value to companion device 108 via a communication channel at interaction 208. The communication channel may be a wireless communication channel, such as Wi-Fi communication (e.g., at approximately 2.4 GHz or 5 GHz) or a near-field communication (NFC) channel (e.g., at approximately 14 MHz). In some embodiments, the communication channel may be an in-band communication channel that can be used for subsequent secure data transmission.

[0037] User device 104 can generate shared data. Optionally or additionally, the shared data can be generated by or based on the user attempting to pair the two devices, specifically user device 104 and / or companion device 108. For example, the shared data may include spoken phrases from the user attempting to pair the two devices, such as alphanumeric phrases, or one or more spectral properties of spoken phrases. User device 104 and companion device 108 can each capture the spoken phrases using their respective microphones. User device 104 and companion device 108 can use speech recognition technology to obtain phrases from the spoken phrases. This document refers to... Figure 5 Examples describing speech recognition technology.

[0038] As another example, the shared data may include biometric information or data of users attempting to pair the two devices, user device 104 and / or companion device 108. User device 104 and companion device 108 may each capture images of the user's eyes. User device 104 and companion device 108 may extract or generate biometric templates, such as iris codes, based on each captured image of the user's eyes. Iris codes can be calculated using various methods. For example, in some embodiments, iris codes may be generated according to an algorithm for iris biometrics developed by John Daugman (see, for example, U.S. Patent No. 5,291,560). For example, an iris code may be based on a convolution of an image of the eye's iris (e.g., in polar coordinates) with a two-dimensional bandpass filter (e.g., a Gabor filter), and the iris code may be represented as a two-bit number (e.g., whether the response to a particular Gabor filter is positive or negative).

[0039] Companion device 108 receives shared data from user device 104. For example, companion device 108 may receive a byte stream containing values ​​via a communication channel. After receiving the shared data, companion device 108 may perform a transformation of the shared data at interaction 212. For example, companion device 108 may generate a visual pattern based on the value at interaction 212. The visual pattern may be a one-dimensional pattern (e.g., a barcode) or a two-dimensional pattern (e.g., a QR code). As another example, companion device 108 may generate audio signals, including audio signals inaudible to the human ear, such as ultrasonic audio signals. Even if the audio signal is inaudible to the human ear, it may be recorded by the microphone of user device 104.

[0040] Companion device 108 can transmit the converted shared data to user device 104 via an out-of-band communication channel at interaction 216. For example, companion device 108 can display a visual pattern at interaction 216 using its display or another optical device (e.g., a light source). As another example, companion device 108 can transmit an audio signal at interaction 216 using its speaker. User device 104 can receive the converted shared data through explicit user interaction. For example, the user of user device 104 can choose when the camera of user device 104 captures a visual pattern or when the microphone of user device 104 records an audio signal. Additionally, user device 104 can receive the converted shared data through non-explicit user interaction. For example, user device 104 can be in pairing mode and automatically receive communication. As another example, user device 104 can use its camera and / or microphone to continuously monitor its surroundings to determine the presence of visual signals on companion device 108 and / or any audio signals transmitted by companion device 108.

[0041] Subsequently, user device 104 can extract shared data from the received transformation at interaction 220. In one example, companion device 108 can display a visual pattern generated based on this value using its display or another optical device (e.g., a light source). The optical device can emit human-perceptible visible or non-visible light (e.g., infrared or ultraviolet light). If companion device 108 is a totem with green light-emitting diodes (LEDs) and red LEDs (e.g., totem 108a), companion device 108 can use both LEDs to display the visual pattern. At interaction 220, user device 104 can capture the visual pattern using its image capture device and extract numbers or values ​​from the visual pattern capture using computer vision techniques. If companion device 108 is a mobile phone with a display (e.g., mobile phone 108b), companion device 108 can display the visual pattern on its display. At interaction 220, user device 104 can capture the displayed visual pattern using its image capture device and extract numbers from the visual pattern capture using computer vision techniques. (References herein) Figure 5 Examples describing computer vision recognition technology.

[0042] At interaction 224, user equipment 104 can verify the consistency between shared data sent to companion device 108 (e.g., shared data sent to companion device 108 at interaction 208) and shared data received and retrieved (e.g., shared data communicated by companion device 108 at interaction 216 and subsequently retrieved by user equipment 104 at interaction 220). If the consistency of the shared data is verified, companion device 108 is authenticated to user equipment 104. For example, if user equipment 104 sends a number to companion device 108 at interaction 208, user equipment 104 can verify at interaction 224 that the number retrieved at interaction 220 is the same number. As another example, the iris codes generated by user equipment 104 and companion device 108 can be sufficiently similar. Due to the variability of iris codes, the consistency of shared data based on iris codes can allow for a certain degree of variability.

[0043] After peer device 108 is authenticated to user device 104, user device 104 and peer device 108 can securely transmit data between each other. In some embodiments, user device 104 may be authenticated to peer device 108. At interaction 228, peer device 108 may generate new shared information or data (e.g., a new shared secret or a new shared key) compared to the shared data sent by user device 104 to peer device 108 at interaction 208. For example, peer device 108 may generate a new value compared to the value sent as a byte stream at interaction 208.

[0044] After generating new shared data at interaction 228, companion device 108 can generate a new transformation of the new shared data at interaction 232, compared to the transformation generated at interaction 212. For example, companion device 108 can generate a new visual pattern (e.g., one-dimensional or two-dimensional code) or a new audio signal at interaction 232 based on the new value. As another example, companion device 108 can generate an audio signal at interaction 232 based on the new value.

[0045] Companion device 108 can transmit new transformations of shared data to user device 104 via an out-of-band communication channel at interaction 236 (e.g., the out-of-band communication channel used at interaction 216). For example, companion device 108 can display a visual pattern at interaction 236 using its display or another optical device (e.g., a light source). As another example, companion device 108 can transmit an audio signal at interaction 236 using its speaker. As discussed with reference to interaction 216, user device 104 can receive transformations of shared data via explicit or implicit user interaction.

[0046] Subsequently, user equipment 104 can extract new shared data from the received new transformation at interaction 240. In one example, at interaction 232, companion device 108 can use its display or another optical device to display a new visual pattern generated based on the new value. After extracting the new shared data from the new transformation received at interaction 240, user equipment 104 can send the new shared data to companion device 108 at interaction 244. For example, user equipment 104 can send a byte stream containing the new value at interaction 244 via a communication channel. The communication channel can be a wireless communication channel, such as Wi-Fi communication or near field communication (NFC) channel. The communication channel can be an in-band communication channel that can be used for subsequent secure data transmission and / or a communication channel used at interaction 208.

[0047] At interaction 248, companion device 108 can verify new shared data sent to user device 104 (e.g., new shared data sent to user device 104 at interaction 236) and new shared data received (e.g., new shared data retrieved by user device 104 at interaction 240 and subsequently sent to companion device 108 at interaction 244). If the consistency of the new shared data is verified, user device 104 is authenticated to companion device 108. For example, if the new transformation sent by companion device 108 to user device 104 at interaction 236 is a number, companion device 104 can verify at interaction 248 whether the same number was received.

[0048] Advantageously, companion device 108 does not need to include an image capture device or a microphone so that user device 104 can be authenticated to companion device 108. For example, companion device 108 may be a simple totem with two LEDs of different colors or with a speaker for out-of-band communication (e.g., totem 108a). Although Figure 2 It is shown that user equipment 104 first authenticates companion device 108, but in some embodiments, user equipment 104 may first authenticate companion device 108.

[0049] Example of device verification using the object of the challenge.

[0050] In some embodiments, companion device 108 may use a challenge object (e.g., challenge text) to authenticate user device 104. Figure 3 This is an interaction diagram illustrating device authentication using challenge text according to one embodiment. A companion device 108 may send information or data objects (e.g., challenge text) to a user device 104 at interaction 304. For example, the companion device 108 may send a byte stream containing challenge text to the user device 104 via a communication channel at interaction 304. The communication channel used to send the data object may be a wireless communication channel, such as Wi-Fi communication or a near-field communication (NFC) channel.

[0051] After sending the challenge text at interaction 304, companion device 108 can generate shared information or data (e.g., a shared secret, a shared key, or a signing key) at interaction 308. For example, companion device 108 can generate shared data randomly. As another example, the shared data can be predetermined.

[0052] After generating shared data at interaction 308, companion device 108 can generate a transformation of the shared data at interaction 312. For example, companion device 108 can generate a visual pattern (e.g., a one-dimensional pattern or a two-dimensional pattern) or an audio signal (e.g., a supersonic audio signal or an audio signal inaudible to the human ear) at interaction 312 based on the shared data generated at interaction 308.

[0053] Companion device 108 can transmit the converted shared data to user device 104 at interaction 316 via an out-of-band communication channel. The out-of-band communication channel can be a visual communication channel for transmitting visual patterns or an audio communication channel for transmitting audio signals. For example, companion device 108 can display visual patterns at interaction 316 using its display or another optical device (e.g., a light source). As another example, companion device 108 can transmit audio signals at interaction 316 using its speaker. As discussed with reference to interaction 216, user device 104 can receive the converted shared data through explicit or implicit user interaction.

[0054] Subsequently, user device 104 can extract shared data from the received transformation at interaction 320. In one example, companion device 108 can display a visual pattern at interaction 316 using its display or another optical device (e.g., two LEDs of different colors). User device 104 can capture an image of the visual pattern and extract shared data from the received visual pattern at interaction 320 using computer vision techniques. As another example, companion device 108 can communicate audio signals at interaction 316 using its speaker. Even if the audio signal is inaudible to the human ear, it can be recorded by the microphone of user device 104. User device 104 can extract shared data from the audio signal at interaction 320.

[0055] User equipment 104 can use shared data at interaction 324 to generate a transformation of the data object. For example, user equipment 104 can use shared data at interaction 324 to generate a hash of the challenge text (e.g., a secure hash algorithm (SHA)-2). As another example, user equipment 104 can use a shared key to encrypt the challenge text at interaction 324.

[0056] User equipment 104 can send the transformed data object generated at interaction 324 to companion device 108 at interaction 328. The communication channel used to send the data object from user equipment 104 to companion device 108 can be a wireless communication channel, such as Wi-Fi communication or near-field communication (NFC) channel. In some implementations, the communication channels at interactions 304 and 328 can be the same. In some embodiments, the communication channel used to send the transformed data object can be an in-band communication channel that can be used for subsequent secure data transmission.

[0057] Companion device 108 receives a transformation of a data object from user device 104. Before or after receiving the transformation, companion device 108 may use shared data to generate the transformation at interaction 332. For example, companion device 108 may use shared data at interaction 332 to generate a hash of the challenge text. As another example, companion device 108 may use a shared key to encrypt the challenge text at interaction 332.

[0058] The companion device 108 can verify the consistency between the shared data generated at interaction 332 and the shared data received from user equipment 104. If the consistency of the shared data is verified, user equipment 104 is authenticated to companion device 108. For example, if the hash received by companion device 108 from user equipment 104 is a number, companion device 104 can verify that the hash generated at interaction 332 is also the same number. After authentication, user equipment 104 and companion device 108 can securely transmit data between each other.

[0059] Advantageously, companion device 108 does not need to include an image capture device or microphone for user device 104 to be authenticated to companion device 108. For example, companion device 108 could be a simple totem with two LEDs of different colors or a speaker for out-of-band communication. Although Figure 3 The illustration shows companion device 108 authenticating user equipment 104, but in some embodiments, user equipment 104 can use... Figure 3 The method shown is used to authenticate companion device 108.

[0060] Example device authentication using shared data generated by peer devices.

[0061] In some embodiments, after receiving shared information or data (e.g., shared secret, shared key, or signature key) generated by the companion device 108 in a converted form, the user equipment 104 can authenticate the companion device 108. Figure 4 This is an interaction diagram illustrating device authentication using shared data generated by companion device 108 according to one embodiment. Companion device 108 may generate shared data (e.g., a shared secret, a shared key, or a signing key) at interaction 404. For example, companion device 108 may generate the signing key randomly. As another example, the shared data may be predetermined. The length of the signing key may be, for example, 64, 128, 256, 512, 1024, or more bits.

[0062] After generating shared data at interaction 404, companion device 108 can perform a transformation of the shared data at interaction 408. For example, companion device 108 can generate a visual pattern (e.g., a one-dimensional pattern or a two-dimensional pattern) or an audio signal (e.g., a supersonic audio signal or an audio signal inaudible to the human ear) at interaction 408 based on the shared data generated at interaction 404. The visual pattern can be a one-dimensional pattern (e.g., a barcode) or a two-dimensional pattern (e.g., a QR code).

[0063] Companion device 108 can transmit the converted shared data to user device 104 at interaction 412 via an out-of-band communication channel. The out-of-band communication channel can be a visual communication channel for transmitting visual patterns or an audio communication channel for transmitting audio signals. For example, companion device 108 can display visual patterns at interaction 412 using its display or another optical device (e.g., a light source). As another example, companion device 108 can transmit audio signals at interaction 412 using its speaker. As discussed with reference to interaction 216, user device 104 can receive the converted shared data through explicit or non-explicit user interaction.

[0064] Subsequently, user device 104 can extract shared data from the received transformation at interaction 416. In one example, companion device 108 can display a visual pattern at interaction 412 using its display or another optical device (e.g., two LEDs of different colors). User device 104 can capture an image of the visual pattern and extract shared data from the received visual pattern at interaction 416 using computer vision techniques. As another example, companion device 108 can use its speaker to communicate audio signals at interaction 412. Even if the audio signal is inaudible to the human ear, it can be recorded by the microphone of user device 104. User device 104 can extract shared data from the audio signal at interaction 416.

[0065] Companion device 108 can perform a conversion of shared data at interaction 408, and user device 104 can extract the shared data at interaction 416 based on a predetermined protocol. For example, if the shared data is the number "57", companion device 108 can cause its green LED to blink 5 times and its red LED to blink 7 times based on the predetermined protocol. User device 104 can extract the number "57" from the blinking pattern based on the predetermined protocol. The predetermined protocol can specify that the shared data is the sum of the number of red LED blinks and ten times the number of green LED blinks.

[0066] The companion device 108 can use the shared data (also referred to as the first shared data) generated by the companion device 108 at interaction 404 to generate a transformation of another shared information or data (also referred to as the second shared information or data) at interaction 420. For example, the companion device 108 can use the signing key generated by the companion device 108 at interaction 404 to encrypt its public key.

[0067] The companion device 108 may send a transformation of other shared data (e.g., second shared data) to the user device 104 at interaction 424. For example, the companion device 108 may send an encrypted public key to the user device 104 at interaction 424. The communication channel used to send the encrypted public key may be a wireless communication channel, such as a Wi-Fi communication or a near-field communication (NFC) channel. In some embodiments, the communication channel used to send the transformation of other shared data at interaction 424 may be an in-band communication channel that can be used for subsequent secure data transmission.

[0068] Companion device 104 can use the shared data extracted at interaction 416 to extract additional shared data (e.g., second shared data) from the transformation of other shared data at interaction 428. User device 104 can authenticate companion device 108 at interaction 432 based on the additional shared data (e.g., second shared data). For example, user device 104 can use the signing key extracted at interaction 416 to decrypt the encrypted public key at interaction 428. User device 104 can authenticate companion device 108 by verifying the consistency between the decrypted public key and companion device 108's public key at interaction 432. User device 104 can obtain companion device 108's public key from a device public key repository. Public key encryption techniques (e.g., Rivest Shamir-Adleman (RSA) encryption) can be used, where each device (e.g., user device 104 and companion device 108) is associated with a public key (widely disseminated) and a private key (kept secure and known only to the parties involved). Public-key cryptography is an example of asymmetric cryptography, in which the key used for encryption is different from the key used for decryption. Other asymmetric cryptographic techniques may be used in other embodiments. After authentication, user equipment 104 and peer equipment 108 can securely transmit data between each other.

[0069] Advantageously, companion device 108 does not need to include an image capture device or a microphone for user equipment 104 to authenticate companion device 108. For example, companion device 108 could be a simple totem with two LEDs of different colors or a speaker for out-of-band communication. Although Figure 4 This illustration shows companion device 108 being authenticated to user equipment 104, but in some embodiments, user equipment 104 can use... Figure 4 The method shown is used to authenticate peer device 108.

[0070] Example of equipment verification method

[0071] Figure 5A flowchart of a device authentication method 500 is shown. This method 500 can be implemented by devices such as head-mounted displays or wearable display systems. The device may generate first shared information or data at block 504. The first shared data may be a shared secret, a shared key, or a signing key. In different implementations, the device generating the first shared data may be different. In some embodiments, user device 104 may generate the first shared data. For example, user device 104 generates the first shared data at interaction 204. User device 104 may send the first shared data to companion device 108 at interaction 208, companion device 108 may perform a transformation of the first shared data at interaction 212, and companion device 108 may send the transformed first shared data to user device 104 at interaction 216. In some embodiments, companion device 108 may generate the first shared data. For example, companion device 108 may generate the first shared data at interaction 308. The device may send the first shared data to another device via a first communication channel.

[0072] At block 508, a device may receive first communication from another device. The first communication may be generated by the device based on first shared data. The first communication may include visual or optical communication (e.g., a one-dimensional optical pattern, such as a barcode, or a two-dimensional optical pattern, such as a quick-response (QR) code), audio communication (e.g., inaudible to the human ear or ultrasound), or any combination thereof. The first shared data may include spoken phrases (e.g., alphanumeric phrases), one or more spectral characteristics of spoken phrases, user biometric information or data (e.g., iris code), or any combination thereof. The first communication may be a transformation of the first shared data and may be received by the device via an out-of-band communication channel, through explicit user interaction, or through non-explicit user interaction. The out-of-band communication channel may be a visual or optical communication channel for transmitting visual patterns, or an audio communication channel for transmitting audio signals. In some embodiments, companion device 108 may receive the first communication. For example, user device 104 may send the first communication to companion device 108 at interaction 208, and companion device 108 may receive the first communication. User device 104 may generate the first communication based on the first shared data. In some embodiments, user device 104 may receive the first communication. For example, companion device 108 can generate a conversion of the first shared data at interaction 312, can communicate the conversion of the first shared data at interaction 316, and user device 104 can receive the conversion of the first shared data.

[0073] At block 512, the device can extract first shared data from the first communication. For example, the first communication may be visual communication, such as a visual pattern. The device can receive images from the visual communication and extract the first shared data using computer vision techniques. As another example, the first shared data may include spoken phrases of users attempting to pair the two devices, such as alphanumeric phrases, or one or more spectral properties of spoken phrases. The device can obtain phrases from spoken phrases using speech recognition techniques. In some embodiments, user device 104 can extract the first shared data from received transformations at interaction 220. In some embodiments, user device 104 can extract the first shared data from received transformations at interaction 320.

[0074] Speech recognition technology can be based on machine learning, such as Long Short-Term Memory (LSTM) and Recurrent Neural Networks (RNNs). Speech recognition technology can also be based on Hidden Markov Models (HMMs), Dynamic Time Warping (DTW) based speech recognition, machine learning, or end-to-end automatic speech recognition. Machine learning techniques for speech recognition can be based on neural networks, such as Long Short-Term Memory (LSTM) Recurrent Neural Networks (RNNs), Time Delay Neural Networks (TDNNs), Deep Feedforward Neural Networks (DNNs), or Recurrent Neural Networks (RNNs).

[0075] Non-limiting examples of computer vision techniques include: Scale Invariant Feature Transform (SIFT), Speed-Up Robust Features (SURF), Oriented Fast and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Feature Surface Method, Lucas Kanade algorithm, Horn-Schunk algorithm, Mean Shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) technique, Sequential Bayesian estimators (such as Kalman filters, Extended Kalman filters, etc.), Bundle adjustment, Adaptive Throttling (and other thresholding techniques), Iterative Nearest Point (ICP), Semi-Global Matching (SGM), Semi-Global Block Matching (SGBM), Feature Point Histogram, various machine learning algorithms (such as Support Vector Machines, k-Nearest Neighbors, Naive Bayes, Neural Networks (including convolutional or deep neural networks) or other supervised / unsupervised models, etc.), and so on.

[0076] Alternatively or additionally, speech recognition and computer vision algorithms can be performed using various machine learning algorithms. Once trained, the machine learning algorithms can be stored by the user device 104. Some examples of machine learning algorithms may include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression), instance-based algorithms (e.g., learned vector quantization), decision tree algorithms (e.g., classification and regression trees), Bayesian algorithms (e.g., Naive Bayes), clustering algorithms (e.g., k-means clustering), association rule learning algorithms (e.g., a-prior algorithms), artificial neural network algorithms (e.g., perceptron), deep learning algorithms (e.g., deep Boltzmann machines or deep neural networks), dimensionality reduction algorithms (e.g., principal component analysis), ensemble algorithms (e.g., cascaded generalization), and / or other machine learning algorithms. In some embodiments, a single model can be customized for a single dataset. For example, the wearable device may generate or store a base model. The base model can be used as a starting point for generating additional models specific to data types (e.g., a specific user in a telepresence session), datasets (e.g., additional image sets of users obtained in a telepresence session), conditional situations, or other variations. In some embodiments, user equipment 104 may be configured to utilize a variety of techniques to generate models for analyzing aggregated data. Other techniques may include using predefined thresholds or data values.

[0077] A device can authenticate another device based on first shared data extracted from the transformation of first shared data. For example, a device can verify the consistency between the first shared data generated in block 504 and the first shared data extracted in block 512. If the consistency of the first shared data is verified, the device is authenticated. In some embodiments, user equipment 104 can verify the consistency between the first shared data sent to companion device 108 and the first shared data subsequently extracted by user equipment 104 and received from companion device 108.

[0078] Optionally, a device (e.g., a first device) can receive second communication from another device (e.g., a second device). The second communication may be generated by the second device based on second shared information or data. The second shared data may be generated by the second device. The first device can extract the second shared data from the received second communication. The first device can send the extracted second shared data to the second device. In some embodiments, the second device can receive the extracted second shared data and authenticate the first device based on the received extracted second shared data. For authentication, the second device can verify the consistency between the extracted received second shared data and the generated second shared data.

[0079] Optionally or additionally, a device (e.g., a first device) may optionally receive the public key of another device (e.g., a second device) and receive the encrypted public key of the second device via a first communication channel. The first communication channel may be a wireless communication channel, such as a Wi-Fi communication channel or a near-field communication channel. The encrypted public key of the second device may be encrypted by the second device using first shared data. The first device may use the first shared data to decrypt the encrypted public key of the second device to obtain the decrypted public key. At block 516, the first device can authenticate the second device by verifying the consistency between the decrypted public key and the public key of the second device.

[0080] Example of device verification between a user device and two companion devices

[0081] Figure 6 Device authentication according to one embodiment is illustrated. User equipment (UD) 104 can authenticate two or more companion devices, such as a first companion device (CD1) 108a and a second companion device (CD2) 108b. User equipment 104 can be an HMD, a wearable display system, a tablet computer, a mobile phone, or a mobile device. Each of the two companion devices 108a, 108b can be a totem (e.g., used as a virtual user input device), an HMD, a wearable display system, a tablet computer, a mobile phone, or a mobile device. For a totem, a user can interact with the totem's input surface (e.g., a touchpad) using their thumb and can use other fingers to hold the totem. References are provided below. Figure 9-14 Examples describing HMDs and totems.

[0082] User equipment 104 can communicate wired or wirelessly with each companion device 108a, 108b via cloud 604 or access point (AP) 606. Communication can be via radio frequency (RF) communication channels (e.g., WiFi, Bluetooth, etc.). For example, user equipment 104 can connect to cloud 604 via communication channel instance 608a. As another example, user equipment 104 can connect to access point (AP) 606, such as a router, via communication channel instance 608b. Access point 612 itself can connect to cloud 604 via communication channel instance 608c. Companion devices 108a, 108b can communicate wired or wirelessly with each other via cloud 604 through communication channels instances 608d, 608e. For example, companion devices 108a, 108b can be close to each other and connected to cloud 604 via access point 606 through communication channels instances 608d, 608e. In one embodiment, for authentication, companion devices 108a, 108b should connect to the same access point 606 to prove they are close to each other. In one embodiment, for authentication, user equipment 104 and companion devices 108a and 108b must connect to the same access point to prove they are close to each other. Communication within the wired or wireless channel bandwidth can represent the primary communication channel (also known as the first communication channel) between the three devices.

[0083] User equipment 104 and the first companion device 108a (or the second companion device 108b) can also be configured to communicate on a second, different communication channel (e.g., an optical channel, an acoustic channel, a near-field communication (NFC) channel, etc.). For example, user equipment 104 can communicate with the first companion device 108a via instance 612a of the second communication channel, and user equipment 104 can communicate with the second companion device 108b via instance 612b of the second communication channel. Similarly, the first companion device 108a and the second companion device 108b can also be configured to communicate on a second, different communication channel. For example, companion devices 108a and 108b can communicate with each other via instance 612c of the second communication channel. This communication can be referred to as out-of-band (OOB) communication (indicated by dashed lines).

[0084] To enable user equipment 104 and companion devices 108a and 108b to authenticate each other and communicate securely, the first companion device 108a may send first shared information or data to user equipment 104 via instance 612a of the second communication channel in a first converted form (e.g., as a visual pattern or audio signal). The first shared data may include an identifier of the first companion device 108a (e.g., the address of the first companion device 108a, such as its Internet Protocol (IP) address or Media Access Control (MAC) address). The first shared data may also include a pair bit (PB) with a specific value (e.g., a value of 1) indicating that the first companion device 108a is available for pairing. The first companion device 108a (or another device) may generate the first shared data in the first converted form. Before, simultaneously with, or after sending the first shared data in the first converted form to user equipment 104, the first companion device 108a may also use instance 608d of the first communication channel to send the first shared data to cloud 604 (e.g., a device on cloud 604), for example, in a non-converted form or a converted form other than the first converted form. In one embodiment, the first shared data sent to cloud 604 may also include pairs of bits.

[0085] Similarly, the second companion device 108b may send the second shared information or data to the user equipment 104 via instance 612b of the second communication channel in a second converted form (e.g., as a visual pattern or audio signal). The second shared data may include an identifier of the second companion device 108b (e.g., the address of the second companion device 108b, such as its IP address or MAC address). The second shared data may also include a pair bit (PB) with a specific value (e.g., a value of 1) indicating that the second companion device 108b is available for pairing. The second companion device 108b (or another device) may generate the second shared data in a second converted form. Before, during, or after sending the second shared data in the second converted form to the user equipment 104, the second companion device 108b may also send the second shared data to the cloud 604 (e.g., a device on the cloud 604) using instance 608e of the first communication channel, for example, in a non-converted form or a converted form other than the second converted form. In one embodiment, the second shared data sent to the cloud 604 may also include a pair bit. In one embodiment, the process of generating first shared data in a first conversion form by the first companion device 108a and the process of generating second shared data in a second conversion form by the second companion device 108b are the same.

[0086] After receiving first shared data in a first converted form from first companion device 108a and second shared data in a second converted form from second companion device 108b, user equipment 104 can send data including the first shared data in the first converted form, the second shared data in the second converted form, and the identifier of user equipment 104 (e.g., the address of user equipment 104, such as the IP address or MAC address of user equipment 104) to cloud 604 (e.g., a device on cloud 604). The device on cloud 604 that receives the first shared data from first companion device 108a, the device on cloud 604 that receives the second shared data from second companion device 108b, and the device on cloud 604 that receives data from user equipment 104 can be the same device.

[0087] The device on cloud 604 can verify the consistency of data received from user equipment 104 and companion devices 108a and 108b. For example, the device on cloud 604 can use first shared data received from first companion device 108a to generate an instance of first shared data in a first transformed form. The device on cloud 604 can verify that the first shared data in the first transformed form received from user equipment 104 is consistent with the first shared data in the first transformed form it generated. As another example, the device on cloud 604 can use second shared data received from second companion device 108b to generate an instance of second shared data in a second transformed form. The device on cloud 604 can verify that the second shared data in the second transformed form received from user equipment 104 is consistent with the second shared data in the second transformed form it generated. In one embodiment, the first shared data includes an identifier of first companion device 108a, and the second shared data includes an identifier of second companion device 108b. The device on cloud 604 can verify the proximity of companion devices 108a and 108b based on the identifiers (e.g., the device's IP address). In one embodiment, the device on cloud 604 can verify the proximity of user device 104 and companion devices 108a and 108b based on an identifier (e.g., the device's IP address). After verifying the consistency of the received data, the device on cloud 604 can authenticate user device 104 and companion devices 108a and 108b.

[0088] After authenticating user equipment 104 and companion devices 108a and 108b, the devices on cloud 604 can generate third shared information or data (e.g., link keys) for communication between user equipment 104 and companion devices 108a and 108b. For example, a device on cloud 604 can send third shared data to first companion device 108a using instance 608d of the first communication channel. A device on cloud 604 can send third shared data to second companion device 108b using instance 608e of the first communication channel. A device on cloud 604 can send third shared data to user equipment 104 using either instance 608a or instance 608b of the first communication channel. After receiving the third shared data, user equipment 104 and companion devices 108a and 108b can encrypt the data sent to each other or decrypt the data received from each other.

[0089] Example device authentication using visual patterns generated by two companion devices.

[0090] In some embodiments, user equipment 104 and two or more companion devices (e.g., first companion device 108a and second companion device 108b) may authenticate or pair with each other using multiple shared information or data. Figure 7 This is an interaction diagram illustrating device authentication between user equipment 104 and the two companion devices 108a, 108b using visual patterns (or audio signals or transformations) generated by the two companion devices 108a, 108b. In one embodiment, the first companion device 108a may use visual communication (e.g., using its display) or audio communication (e.g., using its speaker) at interaction 704 to indicate its pairable status. Upon receiving the pairable status, user equipment 104 may initiate part of its authentication process for the first companion device 108a.

[0091] The first companion device 108a may generate an information or data object, such as a first visual pattern (FVP), at interaction 708. The first companion device 108a may generate the first visual pattern using its address (e.g., its Internet Protocol (IP) address or Media Access Control (MAC) address) or its identifier. In one embodiment, the first companion device 108a may include one or more addresses or identifiers. The first companion device 108a may display (or otherwise communicate using a non-visual pattern or signal) the first visual pattern at interaction 712. Such display of the first visual pattern may be referred to as out-of-band (OOB) communication. The user equipment 104 may read (or receive) the first visual pattern (visual or non-visual pattern or signal) at interaction 716. For example, the user equipment 104 may capture the first visual pattern using its image capture device. After reading the first visual pattern at interaction 716, the user equipment 104 may store the first visual pattern in its memory. For example, the user equipment 104 may temporarily store the first visual pattern until the authentication process is complete.

[0092] The first companion device 108a may send data including the address of the first companion device 108a and a pair of bits (PB) with a specific value (e.g., a value of 1) to the cloud 604 (e.g., a device on the cloud 604) at interaction 720. In one embodiment, the first companion device 108a may send data to the device on the cloud 604 at interaction 720 before displaying the first visual pattern at interaction 712.

[0093] In one embodiment, the second companion device 108b may use visual communication (e.g., using its display) or audio communication (e.g., using its speaker) at interaction 724 to indicate its pairable status. Upon receiving the pairable status, the user equipment 104 may initiate part of its authentication process for the second companion device 108b. The second companion device 108b may generate an information or data object, such as a second visual pattern (SVP), at interaction 728. The second companion device 108b may generate the second visual pattern using its address (e.g., its IP address or MAC address) or its identifier. In one embodiment, the second companion device 108b may include one or more addresses or identifiers.

[0094] The second companion device 108b may display (or otherwise communicate using non-visual patterns or signals) a second visual pattern at interaction 732. This display of the second visual pattern may be referred to as out-of-band (OOB) communication. User equipment 104 may read (or receive) the second visual pattern (visual or non-visual patterns or signals) at interaction 736. For example, user equipment 104 may capture the second visual pattern using its image capture device. After reading the second visual pattern at interaction 736, user equipment 104 may store the second visual pattern in its memory. For example, user equipment 104 may temporarily store the second visual pattern until the authentication process is complete. The second companion device 108b may send data including the address of the second companion device 108b and a pair of bits (PB) of a specific value to cloud 604 (e.g., a device on cloud 604) at interaction 740. In one embodiment, the second companion device 108b may send data to a device on cloud 604 at interaction 740 before displaying the second visual pattern at interaction 732. In one embodiment, the process of generating the first visual pattern implemented by the first companion device 108a and the process of generating the second visual pattern implemented by the second companion device 108b are the same.

[0095] After reading the first visual pattern displayed by the first companion device 108a and the second visual pattern displayed by the second companion device 108b, the user equipment 104 can send data including the first visual pattern, the second visual pattern, and the address or identifier of the user equipment 104 (e.g., the IP address or MAC address of the user equipment 104) to the cloud 604 (e.g., a device on the cloud 604) at interaction 744. The device on the cloud 604 receiving data from the first companion device 108a, the device on the cloud 604 receiving data from the second companion device 108b, and the device on the cloud 604 receiving data from the user equipment 104 can be the same device on the cloud 604.

[0096] The device on cloud 604 can verify the consistency of data received from user equipment 104 and companion devices 108a and 108b at interaction 748. For example, the device on cloud 604 can generate a first visual pattern after interaction 720 using the address of the first companion device 108a received from the first companion device 108a. The device on cloud 604 can verify that the first visual pattern received from user equipment 104 after interaction 744 is consistent with the first visual pattern it generated. As another example, the device on cloud 604 can receive a first visual pattern from the first companion device 108a at interaction 720. The device on cloud 604 can verify that the first visual pattern received from user equipment 104 after interaction 744 is consistent with the first visual pattern received from the first companion device 108a after interaction 720.

[0097] The device on cloud 604 can generate a second visual pattern after interaction 740 using the address of the second companion device 108b received from the second companion device 108b. The device on cloud 604 can verify that the second visual pattern received from user device 104 after interaction 744 is consistent with the second visual pattern it generated. As another example, the device on cloud 604 can receive a second visual pattern from the second companion device 108b at interaction 740. The device on cloud 604 can verify that the second visual pattern received from user device 104 after interaction 744 is consistent with the second visual pattern received from the second companion device 108b after interaction 740.

[0098] After authenticating user equipment 104 and companion devices 108a and 108b, the devices on cloud 604 can generate a link key (LK) at interaction 752. User equipment 104 and companion devices 108a and 108b can use the link key to communicate securely with each other. The devices on cloud 604 can send the link key to the second companion device 108b at interaction 756a. The devices on cloud 604 can send the link key to the first companion device 108a at interaction 756b. The devices on cloud 604 can send the link key to user equipment 104 at interaction 756c. After receiving the link key, user equipment 104 and companion devices 108a and 108b can use the link key to encrypt data sent to each other or decrypt data received from each other. In one embodiment, some of the above interactions can be repeated at fixed intervals (e.g., every second, minute, hour, or more) to generate new link keys. Therefore, user equipment 104 and companion devices 108a and 108b can use the new link key to communicate securely with each other.

[0099] Example device authentication using visual schemas generated by one or more devices in the cloud.

[0100] In some embodiments, user equipment 104 and two or more companion devices (e.g., first companion device 108a and second companion device 108b) can authenticate or pair with each other using multiple shared information or data generated by the device portion on cloud 604. Figure 8 This is an interaction diagram illustrating device authentication between user equipment 104 and two companion devices 108a, 108b using visual patterns at least partially generated by cloud 604 (e.g., one or more devices on cloud 604). In one embodiment, the first companion device 108a may indicate its pairable status at interaction 804 using visual communication (e.g., using its display) or audio communication (e.g., using its speaker). Upon receiving the pairable status, user equipment 104 may initiate part of its authentication process for the first companion device 108a.

[0101] At interaction 808, the first companion device 108a may send data including the address of the first companion device 108a and a pair bit (PB) of a specific value (e.g., a value of 1) to the cloud 604 (e.g., a device on the cloud 604), indicating that the first companion device 108a is available for pairing. After receiving the address and pair bit of the first companion device 108a, the device on the cloud 604 may generate a first pattern (FP) at interaction 812. In one embodiment, the device on the cloud 604 may generate the first pattern using the address of the first companion device 108a. For example, the first pattern may be a hash of the address of the first companion device 108a (e.g., a secure hash algorithm (SHA)-2). The device on the cloud 604 may send the first pattern to the device 108a of the first companion device 108a at interaction 816.

[0102] The first companion device 108a may generate an information or data object, such as a first visual pattern (FVP), at interaction 820. The first companion device 108a may generate a first visual pattern using a received first pattern. In one embodiment, the first companion device 108a may communicate the first visual pattern using an out-of-band (OOB) communication channel. For example, the first companion device 108a may display (or otherwise communicate using non-visual patterns or signals) the first visual pattern at interaction 824. The user device 104 may read the first visual pattern (or receive a visual or non-visual pattern or signal corresponding to the first visual pattern) at interaction 828. For example, the user device 104 may capture the first visual pattern using its image capture device. After reading the first visual pattern at interaction 824, the user device 104 may store the first visual pattern in its memory. For example, the user device 104 may temporarily store the first visual pattern until the authentication process is complete.

[0103] In one embodiment, the second companion device 108b may indicate its pairability status at interaction 832 using visual communication (e.g., using its display) or audio communication (e.g., using its speaker). Upon receiving the pairability status, user equipment 104 may initiate part of its authentication process for the second companion device 108b. The second companion device 108b may send data at interaction 836 including its address and a pair bit (PB) of a specific value, indicating that it is available for pairing with cloud 604 (e.g., a device on cloud 604). Upon receiving the address and pair bit of the second companion device 108b, the device on cloud 604 may generate a second pattern (SP) at interaction 840. In one embodiment, the device on cloud 604 may generate the second pattern using the address of the second companion device 108b. For example, the second pattern may be a hash of the address of the second companion device 108b (e.g., a secure hash algorithm (SHA)-2). The device on cloud 604 may send the second pattern to the second companion device 108b at interaction 844.

[0104] The second companion device 108b may generate an information or data object, such as a second visual pattern (SVP), at interaction 848. The second companion device 108a may generate a second visual pattern using the received second pattern. In one embodiment, the second companion device 108b may communicate the second visual pattern using an out-of-band (OOB) communication channel. For example, the second companion device 108b may display (or otherwise communicate using non-visual patterns or signals) the second visual pattern at interaction 852. The user device 104 may read the second visual pattern (or receive a visual or non-visual pattern or signal corresponding to the second visual pattern) at interaction 856. For example, the user device 104 may capture the second visual pattern using its image capture device. After reading the second visual pattern at interaction 856, the user device 104 may store the second visual pattern in its memory. For example, the user device 104 may temporarily store the second visual pattern until the authentication process is complete.

[0105] After reading the first visual pattern displayed by the first companion device 108a at interaction 824 and the second visual pattern displayed by the second companion device 108b at interaction 852, the user equipment 104 can send data including the first visual pattern, the second visual pattern, and the address or identifier of the user equipment 104 (e.g., the IP address or MAC address of the user equipment 104) to the cloud 604 (e.g., a device on the cloud 604) at interaction 856. The device on the cloud 604 receiving data from the first companion device 108a, the device on the cloud 604 receiving data from the second companion device 108b, and the device on the cloud 604 receiving data from the user equipment 104 can be the same device on the cloud 604.

[0106] The device on cloud 604 can verify the consistency of data received from user equipment 104 and companion devices 108a and 108b at interaction 864. For example, the device on cloud 604 can generate a first visual pattern using a first pattern it generates. The device on cloud 604 can verify that the first visual pattern received from user equipment 104 is consistent with its generated first visual pattern. The device on cloud 604 can generate a second visual pattern using a second pattern it generates. The device on cloud 604 can verify that the second visual pattern received from user equipment 104 is consistent with its generated second visual pattern.

[0107] After authenticating user equipment 104 and companion devices 108a and 108b, the devices on cloud 604 can generate a link key (LK) at interaction 868. User equipment 104 and companion devices 108a and 108b can use the link key to communicate securely with each other. The devices on cloud 604 can send the link key to the second companion device 108b at interaction 872a. The devices on cloud 604 can send the link key to the first companion device 108a at interaction 872b. The devices on cloud 604 can send the link key to user equipment 104 at interaction 872c. After receiving the link key, user equipment 104 and companion devices 108a and 108b can use the link key to encrypt data sent to each other or decrypt data received from each other. In one embodiment, some of the above interactions can be repeated at fixed intervals (e.g., every second, minute, hour, or more) to generate new link keys. Therefore, user equipment 104 and companion devices 108a and 108b can use the new link key to communicate securely with each other.

[0108] Augmented Reality Scene Examples

[0109] Modern computing and display technologies have facilitated the development of so-called "virtual reality" or "augmented reality" experience systems, in which digitally reproduced images or portions thereof are presented to the user in a way that appears real or can be perceived as real. Virtual reality (VR) scenarios typically involve the presentation of digital or virtual image information without transparency to other actual, real visual input; augmented reality (AR) scenarios typically involve the representation of digital or virtual image information as an enhancement to the visualization of the real world surrounding the user; or mixed reality (MR) scenarios typically involve the fusion of the real and virtual worlds to create new environments where physical and virtual objects coexist and interact in real time. The human visual perception system has proven highly complex, and generating rich virtual image elements that promote a comfortable, natural feeling from the presentation of image elements in other virtual or real worlds using VR, AR, or MR technologies has proven challenging. The systems and methods disclosed in this paper address various challenges associated with VR, AR, and MR technologies.

[0110] Figure 9 An illustration depicting an augmented reality scene viewed by a human, featuring specific virtual reality objects and specific real-world objects. Figure 9 An augmented reality scene 900 is depicted, in which the AR user sees a park-like scene 910 in the real world, characterized by people, trees, buildings, and a concrete platform 920 in the background. In addition to these items, the AR user also perceives that he "sees" a robot statue 930 standing on the real-world platform 920, and a cartoon-like avatar 940 (e.g., Bumblebee) flying by, which appears to be an anthropomorphic version of Bumblebee, although these elements do not exist in the real world.

[0111] To enable 3D displays to produce a realistic sense of depth, more specifically, a simulated sense of surface depth, each point in the display's field of view should ideally generate an accommodation response corresponding to its virtual depth. If the accommodation response to a display point does not correspond to the virtual depth of that point as determined by convergence and stereoscopic depth cues, the human eye may experience accommodation conflict, leading to unstable imaging, harmful eye strain, headaches, and, in the absence of accommodation information, almost no surface depth.

[0112] VR, AR, and MR experiences can be provided by display systems with displays that present viewers with images corresponding to multiple depth planes. For each depth plane, the image can be different (e.g., providing a slightly different representation of a scene or object) and can be focused separately by the viewer's eye, thereby facilitating depth cues to the user based on the eye's accommodation required to focus different image features of a scene located on different depth planes, and / or based on observing image features that are out of focus on different depth planes. Such depth cues provide a credible perception of depth, as discussed elsewhere in this paper. To generate or enhance VR, AR, and MR experiences, display systems can use biometric information to augment these experiences.

[0113] Examples of wearable display systems

[0114] Figure 10An example of a wearable display system 1000 is shown, which can be used to present VR, AR, or MR experiences to a wearer or viewer 1004. The wearable display system 1000 can be programmed to perform any of the applications or embodiments described herein. The display system 1000 includes a display 1008 and various mechanical and electronic modules and systems supporting the functionality of the display 1008. The display 1008 may be coupled to a frame 1012, which may be worn by a user, wearer, or viewer 1004 and configured to position the display 1008 in front of the wearer 1004. The display 1008 may be a light field display. In some embodiments, a speaker 1016 is coupled to the frame 1012 and positioned near the user's ear canal. In some embodiments, another speaker (not shown) is positioned near the user's other ear canal to provide stereo / shapeable sound control. Display 1008 is operatively coupled 1020 (e.g., via a wired lead or wireless connection) to local data processing module 1024, which can be installed in various configurations, such as being fixedly attached to frame 1012, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably connected to user 1004 (e.g., in a backpack configuration or a belt-coupled configuration).

[0115] Frame 1012 may have one or more cameras attached or mounted to it to obtain images of the wearer's eyes. In one embodiment, the camera may be mounted on frame 1012 in front of the wearer's eyes so that the eyes can be imaged directly. In other embodiments, the camera may be mounted along a bar of frame 1012 (e.g., near the wearer's ear). In such an embodiment, display 1008 may be coated with a material that reflects light from the wearer's eyes to the camera. The light may be infrared light, as iris features are prominent in infrared images.

[0116] The local processing and data module 1024 may include a hardware processor and non-transitory digital memory, such as non-volatile memory (e.g., flash memory), both of which can be used to assist in data processing, caching, and storage. Data may include (a) data captured from sensors such as image capture devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, and / or gyroscopes (e.g., operatively coupled to frame 1012 or otherwise connected to user 1004); and / or (b) data acquired and / or processed using remote processing module 1028 and / or remote data repository 1032, possibly for transmission to display 1008 after processing or retrieval. The local processing and data module 1024 may be operatively coupled to remote processing module 1028 and remote data repository 1032 via communication links 1036 and / or 1040 (e.g., via wired or wireless communication links), such that these remote modules 1028, 1032 are available as resources for the local processing and data module 1024. The image capture device can be used to capture eye images used in eye image processing. Furthermore, the remote processing module 1028 and the remote data repository 1032 can be operatively coupled to each other.

[0117] In some embodiments, the remote processing module 1028 may include one or more processors configured to analyze and process data and / or image information, such as video information captured by an image capture device. The video data may be stored locally in the local processing and data module 1024 and / or the remote data repository 1032. In some embodiments, the remote data repository 1032 may include a digital data storage device that is available via an additional network configuration in an Internet or "cloud" resource configuration. In some embodiments, storing all data and performing all computations in the local processing and data module 1024 allows for completely autonomous use from the remote module.

[0118] In some implementations, local processing and data module 1024 and / or remote processing module 1028 are programmed to perform embodiments of the systems and methods described herein. The image capture device can capture video for a specific application (e.g., video of a wearer's eyes for an eye-tracking application or video of a wearer's hands or fingers for a gesture recognition application). The video can be analyzed by one or both of the processing modules 1024 and 1028. In some cases, offloading at least part of the iris code generation to a remote processing module (e.g., in the "cloud") can improve computational efficiency or speed. Parameters of the systems and methods disclosed herein can be stored in data modules 1024 and / or 1028.

[0119] The results of the analysis can be used by one or both of the processing modules 1024 and 1028 for additional operations or processing. For example, in various applications, the wearable display system 1000 can use biometric recognition, eye tracking, recognition, or classification of gestures, objects, postures, etc. For example, the wearable display system 1000 can analyze captured video of the wearer 1004's hands and recognize gestures made by the wearer's hands (e.g., picking up a real or virtual object, indicating agreement or disagreement (e.g., "thumbs up" or "thumbs down"), etc.) and the wearable display system.

[0120] The human visual system is complex, and providing realistic depth perception is a challenge. Without being bound by theory, it is believed that an observer might perceive an object as three-dimensional due to the combination of vergence and accommodation. The relative convergence and divergence movements of the eyes (e.g., the rolling motion of the pupils toward or away from each other to focus the eye's gaze on an object) are closely related to the focusing (or "accommodation") of the eye's lenses. Under normal circumstances, changing the focal length of the eye's lenses, or adjusting the eyes to shift focus from one object to another at a different distance, will automatically result in a change in convergence-divergence matching at the same distance, under a relationship known as "accommodation-convergence reflex." Under normal circumstances, changes in convergence will trigger corresponding changes in accommodation. Display systems that provide better accommodation and convergence matching can create more realistic or comfortable simulations of three-dimensional images.

[0121] Figure 11 This illustrates various aspects of a method for simulating 3D images using multiple depth planes. (Refer to...) Figure 11Objects at different distances along the z-axis from eyes 1102 and 1104 are adjusted by eyes 1102 and 1104 such that these objects are in focus. Eyes 1102 and 1104 assume a specific adjustment state to focus objects at different distances along the z-axis. Therefore, it can be said that a specific adjustment state is associated with a specific one of the depth planes 1106 having an associated focal length, such that when the eye is in the adjustment state of that depth plane, an object or part of an object in that depth plane is in focus. In some embodiments, the three-dimensional image can be simulated by providing different representations of the image for each of eyes 1102 and 1104, and can also be simulated by providing different representations of the image corresponding to each depth plane. Although shown separately for clarity, it will be understood that, for example, the fields of view of eyes 1102 and 1104 may overlap as the distance along the z-axis increases. Furthermore, although shown as planes for ease of illustration, it should be understood that the contours of the depth planes may be curved in physical space such that all features in the depth planes are in focus with the eye in a specific adjustment state. Without theoretical constraints, it is believed that the human eye can typically interpret a finite number of depth planes to provide depth perception. Therefore, a highly reliable simulation of perceived depth can be achieved by providing the eye with different representations of the image corresponding to each of these finite number of depth planes.

[0122] Waveguide stack-up component example

[0123] Figure 12 An example of a waveguide stack for outputting image information to a user is shown. The display system 1200 includes a waveguide stack or layered waveguide assembly 1205, which is used to provide three-dimensional perception to the eye 1210 or brain using multiple waveguides 1220, 1222, 1224, 1226, 1228. In some embodiments, the display system 1200 may correspond to... Figure 10 System 1000, Figure 12 Some parts of the system 1000 are illustrated schematically in more detail. For example, in some embodiments, the waveguide assembly 1205 may be integrated into... Figure 10 In monitor 1008.

[0124] Continue to refer to Figure 12 The waveguide assembly 1205 may further include a plurality of features 1230, 1232, 1234, and 1236 between the waveguides. In some embodiments, features 1230, 1232, 1234, and 1236 may be lenses. In some embodiments, features 1230, 1232, 1234, and 1236 may not be lenses. Instead, they may be spacers (e.g., cladding and / or structures for forming air gaps).

[0125] Waveguides 1220, 1222, 1224, 1226, 1228 and / or multiple lenses 1230, 1232, 1234, 1236 can be configured to transmit image information to the eye with different levels of wavefront curvature or light divergence. Each waveguide level can be associated with a specific depth plane and can be configured to output image information corresponding to that depth plane. Image injection devices 1240, 1242, 1244, 1246, 1248 can be used to inject image information into waveguides 1220, 1222, 1224, 1226, 1228, wherein each waveguide can be configured to distribute incident light across its respective waveguide for output to the eye 1210. Light exits the output surfaces of image injection devices 1240, 1242, 1244, 1246, and 1248 and is injected into the corresponding input edges of waveguides 1220, 1222, 1224, 1226, and 1228. In some embodiments, a single beam (e.g., a collimated beam) may be injected into each waveguide to output a fully clonal collimated beam field toward the eye 1210 at a specific angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0126] In some embodiments, image injection devices 1240, 1242, 1244, 1246, and 1242 are separate displays that generate image information for injection into corresponding waveguides 1220, 1222, 1224, 1226, and 1228, respectively. In some other embodiments, image injection devices 1240, 1242, 1246, 1246, and 1248 are the outputs of a single multiplexed display that can, for example, deliver image information to each of the image injection devices 1240, 1242, 1244, 1246, and 1248 via one or more optical guides (e.g., optical fiber cables).

[0127] Controller 1250 controls the operation of the stacked waveguide assembly 1205 and image injection devices 1240, 1242, 1244, 1246, and 1248. In some embodiments, controller 1250 includes adjusting the timing of image information provided to waveguides 1220, 1222, 1224, 1226, and 1228 and providing programming (e.g., instructions in a non-transitory computer-readable medium). In some embodiments, controller 1250 may be a single monolithic device or a distributed system connected via wired or wireless communication channels. In some embodiments, controller 1250 may be a processing module 1024 or 1028 (e.g., ...). Figure 10As shown in the diagram, this is part of the display system 1200. In some embodiments, the controller may communicate with an inward imaging system 1252 (e.g., a digital camera), an outward imaging system 1254 (e.g., a digital camera), and / or a user input device 1266. The inward imaging system 1252 (e.g., a digital camera) may be used to capture images of the eye 1210 to, for example, determine the size and / or orientation of the pupil of the eye 1210. The outward imaging system 1254 may be used to image a portion of the world 1256. A user may input commands to the controller 1250 via the user input device 1266 to interact with the display system 1200.

[0128] Waveguides 1220, 1222, 1224, 1226, and 1228 can be configured to propagate light in each respective waveguide via total internal reflection (TIR). Waveguides 1220, 1222, 1224, 1226, and 1228 can each be planar or have another shape (e.g., curved), having a main top surface and a bottom surface, and an edge extending between these main top and bottom surfaces. In the illustrated configuration, waveguides 1220, 1222, 1224, 1226, and 1228 can each include light extraction optics 1260, 1262, 1264, 1266, and 1268, configured to extract light from the waveguide by redirecting the light, propagating it within the respective waveguide, and exiting the waveguide, thereby outputting image information to the eye 1210. The extracted light can also be referred to as externally coupled light, and the light extraction optics can also be referred to as externally coupled optics. At the location where the light propagating in the waveguide is incident on the light redirection element, the waveguide outputs the extracted beam. The light extraction optical elements (1260, 1262, 1264, 1266, 1268) may be, for example, reflective and / or diffractive optical features. Although shown as disposed on the bottom main surface of waveguides 1220, 1222, 1224, 1226, 1228 for ease of description and clarity of the drawings, in some embodiments, the light extraction optical elements 1260, 1262, 1264, 1266, 1268 may be disposed on the top and / or bottom main surfaces, and / or may be disposed directly within the volume of waveguides 1220, 1222, 1224, 1226, 1228. In some embodiments, the light extraction optical elements 1260, 1262, 1264, 1266, 1268 may be formed in a material layer attached to the transparent substrate used to form waveguides 1220, 1222, 1224, 1226, 1228. In some other embodiments, waveguides 1220, 1222, 1224, 1226, and 1228 may be a single piece of material, and light extraction optical elements 1260, 1262, 1264, 1266, and 1268 may be formed on the surface and / or inside the single piece of material.

[0129] Continue to refer to Figure 12As discussed herein, each waveguide 1220, 1222, 1224, 1226, 1228 is configured to output light to form an image corresponding to a specific depth plane. For example, the waveguide 1220 closest to the eye can be configured to deliver collimated light injected into it to the eye 1210. The collimated light may represent the optical infinity focal plane. The next waveguide 1222 can be configured to emit collimated light that passes through a first lens 1230 (e.g., a negative lens) before reaching the eye 1210. The first lens 1230 can be configured to produce a slight convex wavefront curvature so that the eye / brain interprets the light from the next waveguide 1222 as light from a first focal plane further inward toward the eye 1210 from optical infinity. Similarly, the third upper waveguide 1224 passes its output light through the first lens 1230 and the second lens 1232 before its output light reaches the eye 1210. The combined optical power of the first and second lenses 1230 and 1232 can be configured to create another increment in wavefront curvature so that the eye / brain interprets light from the third waveguide 1224 as light from a second focal plane that is more inward toward the person from optical infinity compared to light from the next waveguide 1222.

[0130] The other waveguide layers (e.g., waveguides 1226, 1228) and lenses (e.g., lenses 1234, 1236) are configured similarly, with the highest waveguide 1228 in the stack transmitting its output through all the lenses between it and the eye, representing the total optical power of the focal plane closest to the eye. To compensate the stack of lenses 1230, 1232, 1234, 1236 when viewing / interpreting light from the world 1256 on the other side of the stacked waveguide assembly 1205, a compensation lens layer 1238 can be positioned on top of the stack to compensate for the total refractive power of the underlying lens stacks 1230, 1232, 1234, 1236. This configuration provides as many perceptible focal planes as possible because of the available waveguide / lens pairs. The focusing aspects of the light extraction optics 1260, 1262, 1264, 1264, 1228 of waveguides 1220, 1222, 1224, 1226, 1228 and lenses 1230, 1232, 1234, 1236 can all be static (e.g., not dynamic or electrically active). In some alternative embodiments, one or both can use electrically active features instead of being dynamic.

[0131] Continue to refer to Figure 12The light extraction optics 1260, 1262, 1264, 1266, and 1268 can be configured to redirect light from their respective waveguides and output the light with appropriate divergence or collimation for a specific depth plane associated with the waveguide. As a result, waveguides with different associated depth planes can have different configurations of light extraction optics that output light with different divergences depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optics 1260, 1262, 1264, 1266, and 1268 can be volumetric or surface features that can be configured to output light at a specific angle. For example, the light extraction optics 1260, 1262, 1264, 1266, and 1268 can be volumetric holograms, surface holograms, and / or diffraction gratings. Optical elements for light extraction, such as diffraction gratings, are described in U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015, the entirety of which is incorporated herein by reference. In some embodiments, features 1230, 1232, 1234, 1236, and 1238 may not be lenses. Instead, they may simply be spacers (e.g., cladding and / or structures for forming air gaps).

[0132] In some embodiments, light extraction optics 1260, 1262, 1264, 1266, and 1268 are diffraction features, or “diffraction optics” (also referred to herein as “DOEs”), that form a diffraction pattern. Preferably, the DOE has a relatively low diffraction efficiency, such that only a portion of the light beam is deflected toward the eye 1210 at each intersection with the DOE, while the remaining light continues to move through the waveguide via total internal reflection. Thus, the light carrying image information is split into multiple associated outgoing beams that exit the waveguide at multiple locations, resulting in a fairly uniform outgoing emission pattern toward the eye 1210 for this particular collimated beam reflected within the waveguide.

[0133] In some embodiments, one or more DOEs can switch between an "on" state where they actively diffract and an "off" state where they do not obviously diffract. For example, a switchable DOE may include a polymer-dispersed liquid crystal layer, wherein the droplets contain diffraction patterns in a host medium, and the refractive index of the droplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not obviously diffract incident light), or the droplets can be switched to a position where the refractive index does not match the host medium (in which case the pattern actively diffracts incident light).

[0134] In some embodiments, the number and distribution of depth planes and / or depth of field can be dynamically changed based on the viewer's pupil size and / or eye orientation. In some embodiments, an inward imaging system 1252 (e.g., a digital camera) can be used to capture images of the eye 1210 to determine the size and / or orientation of the pupil of the eye 1210. In some embodiments, the inward imaging system 1252 can be attached to a frame 1012 (e.g., a digital camera). Figure 10 (as shown), and can be in electrical communication with processing modules 1024 and / or 1028 that process image information from the inward imaging system 1252 to determine, for example, the pupil diameter or eye orientation of user 1004.

[0135] In some embodiments, the inward imaging system 1252 (e.g., a digital camera) can observe user movements, such as eye movements and facial movements. The inward imaging system 1252 can be used to capture images of the eyes 1210 to determine the size and / or orientation of the pupils of the eyes 1210. The inward imaging system 1252 can be used to acquire images for determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris recognition). Images acquired by the inward imaging system 1252 can be analyzed to determine the user's eye pose and / or mood, which the display system 1200 can use to determine which audio or video content should be presented to the user. The display system 1200 can also use sensors such as an inertial measurement unit (IMU), accelerometer, gyroscope, etc., to determine head pose (e.g., head position or head orientation). Head pose can be used alone or in combination with eye pose to interact with the stem track and / or present audio content.

[0136] In some embodiments, a camera may be used for each eye to individually determine the pupil size and / or orientation of each eye, thereby allowing the presentation of image information to each eye to dynamically adapt to that eye. In some embodiments, at least one camera may be used for each eye to independently determine the pupil size and / or eye pose of each eye, thereby allowing the presentation of image information to each eye to dynamically adapt to that eye. In some other embodiments, it is determined and assumed that the pupil diameter and / or orientation of only a single eye 1210 (e.g., using only a single camera per pair of eyes) are similar to the binoculars of viewer 1004.

[0137] For example, depth of field may be inversely proportional to the viewer's pupil size. As a result, as the viewer's pupil size decreases, the depth of field increases, making a plane that was previously indistinguishable because its position was beyond the eye's focal depth discernible, and appearing more in focus as pupil size decreases and depth of field increases accordingly. Similarly, the number of inter-depth planes used to present different images to the viewer can decrease as pupil size decreases. For example, without adjusting the eye's accommodation from one depth plane to another, the viewer may not be able to clearly perceive the details of the first and second depth planes at one pupil size. However, these two depth planes can be sufficiently in focus for the user simultaneously at another pupil size without changing accommodation.

[0138] In some embodiments, the display system may vary the number of waveguides receiving image information based on the determination of pupil size and / or orientation, or based on receiving electrical signals indicating a specific pupil size and / or orientation. For example, if the user's eye cannot distinguish two depth planes associated with two waveguides, the controller 1250 may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this reduces the processing burden on the system, thereby increasing the system's responsiveness. In embodiments where the DOE used for the waveguides is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0139] In some embodiments, it may be desirable for an outgoing light beam to have a diameter smaller than that of the viewer's eye. However, satisfying this condition can be challenging given the variation in the viewer's pupil size. In some embodiments, the condition is satisfied over a wide range of pupil sizes by varying the size of the outgoing light beam in response to a determination of the viewer's pupil size. For example, the size of the outgoing light beam may also decrease as the pupil size decreases. In some embodiments, the size of the outgoing light beam can be varied using a variable aperture.

[0140] Display system 1200 may include an outward imaging system 1254 (e.g., a digital camera) that images a portion of world 1256. This portion of world 1256 may be referred to as the field of view (FOV), and imaging system 1254 is sometimes referred to as an FOV camera. The entire area that can be viewed or imaged by viewer 1004 may be referred to as the field of observation (FOR). FOR may include a solid angle of 4π steradian degrees around display system 1200. In some implementations of display system 1200, FOR may include substantially all solid angles around user 1004 of display system 1200, as user 1004 may move their head and eyes to observe objects around the user (in front, behind, above, below, or to the side of the user). Images obtained from outward imaging system 1254 can be used to track user gestures (e.g., hand or finger gestures), detect objects in world 1256 in front of the user, etc.

[0141] Display system 1200 may include user input device 1266, through which a user can input commands to controller 1250 to interact with display system 400. For example, user input device 1266 may include a trackpad, touchscreen, joystick, multi-degrees-of-freedom (DOF) controller, capacitive sensing device, game controller, keyboard, mouse, D-pad, wand, haptic device, totem (e.g., used as a virtual user input device), etc. In some cases, the user can use a finger (e.g., thumb) to press or slide a touch-sensitive input device to provide input to display system 1200 (e.g., to provide user input to the user interface provided by display system 1200). During use of display system 1200, user input device 1266 can be held by the user's hand. User input device 1266 can communicate with display system 1200 via wired or wireless communication.

[0142] Figure 13An example of an outgoing beam from a waveguide is shown. One waveguide is illustrated, but it is understood that other waveguides in waveguide assembly 1205 can operate similarly, where waveguide assembly 1205 comprises multiple waveguides. Light 1305 is injected into waveguide 1220 at its input edge 1310 and propagates within waveguide 1220 via total internal reflection (TIR). At the point where light 1305 is incident on diffractive optical element (DOE) 1260, a portion of the light exits the waveguide as an outgoing beam 1315. Outgoing beams 1315 are shown as substantially parallel, but they can also be redirected to propagate at an angle (e.g., forming a diverging outgoing beam) towards eye 1210, depending on the depth plane associated with waveguide 1220. Understandably, a substantially parallel outgoing beam can direct a waveguide with light-extracting optics that couples light out to form an image that appears to be positioned on a depth plane at a large distance (e.g., optical infinity) from the eye 1210. Other waveguides or other assemblies of light-extracting optics can output a more divergent outgoing beam pattern, which would require the eye 1210 to adjust to a closer distance to focus it on the retina and would be interpreted by the brain as light closer to the eye 1210 than optical infinity.

[0143] Figure 14 Another example of a display system 1200 is shown, which includes a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The display system 1200 can be used to generate multifocal volumes, images, or light fields. The display system 1200 may include one or more principal planar waveguides 1404. Figure 14 (Only one is shown in the diagram) and one or more do1408s associated with each of at least some of the main waveguides 1404. The planar waveguide 1404 may be similar to the reference. Figure 12 Waveguides 1220, 1222, 1224, 1226, and 1228 are discussed. The optical system can employ distributed waveguide devices to... (The sentence is incomplete and requires further context to translate accurately.) Figure 14 The distributed waveguide device relays the light along a first axis (e.g., the Y-axis) and extends the effective exit pupil of the light. For example, the distributed waveguide device may include a distributed planar waveguide 1412 and at least one DOE 1416 (indicated by a double-dotted line) associated with the distributed planar waveguide 1412. The distributed planar waveguide 1412 may be similar to or identical to the main planar waveguide 1404 in at least some respects, and has a different orientation than the main planar waveguide 1404. Similarly, at least one DOE 1416 may be similar to or identical to DOE 1408 in at least some respects. For example, the distributed planar waveguide 1412 and / or DOE 1416 may be composed of the same material as the main planar waveguide 1404 and / or DOE 1408, respectively. Figure 14The optical system shown can be integrated into Figure 10 The wearable display system 1000 shown is described.

[0144] Relayed and exit pupil-expanded light is optically coupled from the distributed waveguide device into one or more master planar waveguides 1404. The master planar waveguides 1404 are along a second axis (preferably orthogonal to the first axis) (e.g., Figure 14 The second axis (horizontal or X-axis) relays the light. It is worth noting that the second axis can be a non-orthogonal axis to the first axis. The main planar waveguide 1404 extends the effective exit path of the light along the second axis (e.g., the X-axis). For example, a distribution planar waveguide 1412 can relay and extend the light along the vertical or Y-axis and transmit that light to the main planar waveguide 1404, which relays and extends the light along the horizontal or X-axis.

[0145] The display system 1200 may include one or more color light sources (e.g., red, green, and blue lasers) 1420, which may be optically coupled to the near end of a single-mode optical fiber 1424. The far end of the optical fiber 1424 may pass through or be received in a hollow tube 1428 of piezoelectric material. The far end protrudes from the tube 1428 as a fixed-free flexible cantilever 1432. The piezoelectric tube 1428 may be associated with four-quadrant electrodes (not shown). For example, the electrodes may be plated on the outer surface, outer periphery, or outer diameter of the tube 1428. A core electrode (not shown) is also located in the core, center, inner periphery, or inner diameter of the tube 1428.

[0146] A driving electronic device 1436, for example electrically coupled via wire 1440, drives opposing electrode pairs to cause the piezoelectric tube 1428 to bend independently on two axes. The protruding distal end of the optical fiber 1424 has a mechanical resonant mode. The resonant frequency can depend on the diameter, length, and material properties of the optical fiber 1424. By vibrating the piezoelectric tube 1428 near the first mechanical resonant mode of the optical fiber cantilever beam 1432, the optical fiber cantilever beam 1432 is vibrated and can sweep large deflections.

[0147] By exciting the resonance of two axes, a bidirectional scan of the tip of the fiber cantilever 1432 is performed in a two-dimensional scan that fills a region. The light emitted from the fiber cantilever 1432 forms an image by modulating the intensity of the light source 1420 in sync with the scan of the fiber cantilever 1432. A description of such an arrangement is provided in U.S. Patent Publication 2014 / 0003762, which is incorporated herein by reference.

[0148] Component 1444 of the optical coupler subsystem collimates the light emitted from the scanning fiber cantilever beam 1432. The collimated light is reflected by reflective surface 1448 into a narrow distributed planar waveguide 1412 containing at least one diffractive optical element (DOE) 1416. The collimated light undergoes total internal reflection perpendicular (relative to) the distributed planar waveguide 1412. Figure 14The light propagates through the TIR (view of the main planar waveguide 1404) and repeatedly intersects with DOE 1416 in doing so. DOE 1416 preferably has low diffraction efficiency. This causes a portion of the light (e.g., 10%) to diffract towards the edge of the larger main planar waveguide 1404 at each intersection with DOE 1416, and a portion of the light continues along its original trajectory along the length of the distribution planar waveguide 1412 via TIR.

[0149] At each intersection with DOE 1416, additional light diffracts toward the entrance of main waveguide 1412. By splitting the incident light into multiple externally coupled groups, the exit pupil of the light is vertically extended by DOE 1416 in the distributed planar waveguide 1412. This vertically extended light coupled out of the distributed planar waveguide 1412 enters the edge of the main planar waveguide 1404.

[0150] Light entering the main waveguide 1404 travels via TIR along the main waveguide 1404 horizontally (relative to) Figure 14 The light propagates through the TIR (transmission in the view of the main waveguide 1404). As the light propagates horizontally along at least a portion of the length of the main waveguide 1404, it intersects with the DOE 1408 at multiple points. The DOE 1408 can be advantageously designed or configured with a phase distribution having a sum of linear and radially symmetric diffraction patterns to produce light deflection and focusing. The DOE 1408 can advantageously have a low diffraction efficiency (e.g., 10%), such that at each intersection of the DOE 1408, only a portion of the beam is deflected toward the eye, while the remaining light continues to propagate through the TIR via the waveguide 1404.

[0151] At each intersection between the propagating light and the DOE 1408, a portion of the light diffracts toward the adjacent surface of the main waveguide 1404, thereby allowing the light to escape the TIR and emerge from the surface of the main waveguide 1404. In some embodiments, the radially symmetrical diffraction pattern of the DOE 1408 additionally imparts a focus level to the diffracted light, both of which shape the optical wavefront of the single beam (e.g., provide curvature) and control the beam at an angle that matches the designed focus level.

[0152] Accordingly, by varying the DOE1408 at different angles and focus levels, and / or by generating different fill patterns at the exit pupil, these different paths can couple light out of the main planar waveguide 1404. Different fill patterns at the exit pupil can be advantageously used to create light field displays with multiple depth planes. Each layer or a set of layers in the waveguide assembly (e.g., three layers) can be used to generate their respective colors (e.g., red, blue, green). Thus, for example, a first set of three adjacent layers can be used to generate red, blue, and green light respectively at a first depth of focus. A second set of three adjacent layers can be used to generate red, blue, and green light respectively at a second depth of focus. Multiple sets can be used to generate full 3D or 4D color image light fields with different depths of focus.

[0153] Additional aspects

[0154] In a first aspect, a head-mounted display system is disclosed. The head-mounted display system includes: a non-transitory computer-readable storage medium storing executable instructions; and a processor communicating with the non-transitory computer-readable storage medium, the processor being programmed by the executable instructions to: receive from a first device a first communication including a first data object, wherein the first communication is generated at least in part based on first shared data, wherein the first device is configured to send the first shared data to a third device; receive from a second device a second communication including a second data object, wherein the second communication is generated at least in part based on the second shared data, wherein the second device is configured to send the second shared data to the third device; send the first data object and the second data object to the third device, wherein the third device is configured to authenticate the head-mounted display system, the first device, and the second device at least in part based on the first shared data from the first device, the second shared data from the second device, the first data object from the head-mounted display system, and the second data object from the head-mounted display system; and receive from the third device a third communication including a third data object, wherein the third data object indicates that the third device has successfully authenticated the head-mounted display system, the first device, and the second device.

[0155] In the second aspect, the head-mounted display system of aspect 1, wherein a third device communicates with the head-mounted display system, the first device, and the second device via a cloud.

[0156] In the third aspect, a head-mounted display system in either aspect 1-2, wherein the processor is further programmed by executable instructions to send an identifier of the head-mounted display system to a third device.

[0157] In a fourth aspect, the head-mounted display system of aspect 4, wherein the third device authenticates the head-mounted display system, the first device, and the second device at least in part based on the identifier of the head-mounted display system.

[0158] In aspect 5, a head-mounted display system in any of aspects 3-4, wherein the third data object includes an encryption key.

[0159] In the sixth aspect, a head-mounted display system of any of aspects 1-5, wherein the first communication is generated by the first device.

[0160] In the seventh aspect, a head-mounted display system of any of aspects 1-6, wherein the second communication is generated by the second device.

[0161] In the eighth aspect, in any of the aspects 1-7, a head-mounted display system, wherein, for authenticating the head-mounted display system, the first device, and the second device, a third device is configured to: generate a first data object based on first shared data; verify the consistency between the first data object generated by the third device and the first data object from the head-mounted display system; generate a second data object based on second shared data; and verify the consistency between the second data object generated by the third device and the second data object from the head-mounted display system.

[0162] In the ninth aspect, a head-mounted display system of any of aspects 1-8, wherein the first communication is generated by a first device at least in part based on first data, wherein the first data is generated by a third device at least in part based on first shared data.

[0163] In the 10th aspect, the head-mounted display system of aspect 9, wherein the second communication is generated by the second device based on at least a portion of the second data, wherein the second data is generated by the third device based on at least a portion of the second shared data.

[0164] In aspect 11, the head-mounted display system of aspect 10, wherein, in order to authenticate the head-mounted display system, the first device, and the second device, the third device is configured to: generate a first data object based on first data; verify the consistency between the first data object generated by the third device and the first data object from the head-mounted display system; generate a second data object based on second data; and verify the consistency between the second data object generated by the third device and the second data object from the head-mounted display system.

[0165] In aspect 12, a head-mounted display system of any of aspects 1-11, wherein the first shared data includes an identifier of a first device, and wherein the second shared data includes an identifier of a second device.

[0166] In aspect 13, the head-mounted display system of aspect 12, wherein the identifier of the first device includes the address of the first device, and wherein the identifier of the second device includes the address of the second device.

[0167] In aspect 14, the head-mounted display system of aspect 13, wherein at least one of the addresses of the first device and the second device includes the Internet Protocol (IP) address of the first device, the Media Access Control (MAC) address of the first device, or a combination thereof.

[0168] In aspect 15, a head-mounted display system of any of aspects 12-14, wherein an identifier of a first device uniquely identifies the first device, and wherein an identifier of a second device uniquely identifies the second device.

[0169] In aspect 16, a head-mounted display system of any of aspects 1-15, wherein at least one of the first shared data and the second shared data includes spoken phrases.

[0170] In aspect 17, the head-mounted display system of aspect 16, wherein the spoken phrases include alphanumeric phrases.

[0171] In aspect 18, a head-mounted display system of any of aspects 1-17, wherein at least one of the first shared data and the second shared data contains one or more spectral features of spoken phrases.

[0172] In aspect 19, a head-mounted display system of any of aspects 1-18, wherein at least one of the first shared data and the second shared data includes the user's biometric data.

[0173] In aspect 20, a head-mounted display system of any of aspects 1-19, wherein at least one of the first and second communications includes optical communication.

[0174] In aspect 21, the head-mounted display system of aspect 20, wherein optical communication includes a one-dimensional pattern, a two-dimensional optical pattern, or a combination thereof.

[0175] In aspect 22, a head-mounted display system of any of aspects 1-21, wherein at least one of the first and second communications includes audio communication.

[0176] In aspect 23, aspect 22's head-mounted display system, in which audio communication is inaudible to the human ear.

[0177] In aspect 24, aspect 22's head-mounted display system, wherein audio communication is ultrasonic.

[0178] In aspect 25, a head-mounted display system of any one of aspects 1-24, wherein at least one of a first communication, a second communication, and a third communication is received via a wireless communication channel.

[0179] In aspect 26, the head-mounted display system of aspect 25, wherein the wireless communication channel includes a Wi-Fi communication channel, a near-field communication (NFC) channel, or a combination thereof.

[0180] In aspect 27, the head-mounted display system of any of aspects 1-26, wherein the first device or the second device includes at least one of another head-mounted display system, a head-mounted display system, a totem, a mobile phone, a tablet computer, a mobile device, or any combination thereof.

[0181] In aspect 28, a head-mounted display system of any of aspects 1-27, wherein the first device and the second device are connected to the third device via an access point.

[0182] In aspect 29, a head-mounted display system of any of aspects 1-28, wherein the first device, the second device, and the head-mounted display system are connected to the third device via an access point.

[0183] In aspect 30, a head-mounted display system of any of aspects 1-29, wherein the processor is further programmed by executable instructions to: notify the user of the head-mounted display system that a third device has successfully authenticated the head-mounted display.

[0184] In aspect 31, the head-mounted display system of aspect 30, wherein, in order to notify the user of the head-mounted display system, the processor is programmed to: use a display to notify the user of the head-mounted display system that a third device has successfully authenticated the head-mounted display.

[0185] In aspect 32, in any of aspects 30-31, the head-mounted display system, wherein in order to notify the user of the head-mounted display system, the processor is programmed to: cause the display to modify the user interface shown to the user to notify the user of the head-mounted display system that a third device has successfully verified the head-mounted display.

[0186] In aspect 33, in any of aspects 30-32, a head-mounted display system, wherein, in order to notify the user of the head-mounted display, the processor is programmed to: use the speaker of the head-mounted display system to notify the user of the head-mounted display that a third device has successfully authenticated the head-mounted display.

[0187] In aspect 34, a wearable display system is disclosed. The wearable display system includes: a display; an image capturing device configured to capture an image of a companion device; a non-transitory computer-readable storage medium configured to store the image of the companion device and executable instructions; and a processor communicating with the image capturing device and the non-transitory computer-readable storage medium, the processor being programmed by the executable instructions to: receive a first image of a first optical pattern displayed by the companion device, captured by the image capturing device, wherein the first optical pattern is generated by the companion device based on first shared data; extract first data from the first optical pattern in the received first image; and authenticate the companion device based on the first data extracted from the first optical pattern.

[0188] In aspect 35, the wearable display system of aspect 34, wherein the processor is further programmed to: generate first shared data; and transmit the first shared data to a companion device via a first communication channel.

[0189] In aspect 36, a wearable display system of any of aspects 34-35, wherein, in order to authenticate companion devices, the processor is programmed to verify the consistency of first data and first shared data.

[0190] In aspect 37, a wearable display system of any of aspects 34-36, wherein the processor is further programmed to: receive a second image of a second optical pattern displayed by a companion device, captured by an image capturing device, wherein the second optical pattern is generated by the companion device based on second shared data, and wherein the second shared data is generated by the companion device; extract second data from the second optical pattern in the received second image; and transmit the second data to the companion device via a first communication channel.

[0191] In aspect 38, the wearable display system of aspect 37, wherein the companion device is programmed to: receive second data from the wearable display system; and authenticate the wearable display system based on the received second data.

[0192] In aspect 39, the wearable display system of aspect 38, wherein, in order to authenticate the wearable display system, the companion device is programmed to: verify the consistency of the received second data and the second shared data.

[0193] In aspect 40, a wearable display system of any of aspects 34-39, wherein the first shared data is generated by a companion device.

[0194] In aspect 41, the wearable display system of any of aspects 34-40, wherein the processor is further programmed to: receive a public key of a companion device; receive an encrypted public key of the companion device via a first communication channel, wherein the encrypted public key of the companion device is encrypted by the companion device using first shared data; and decrypt the encrypted public key of the companion device using the first shared data to obtain a decrypted public key.

[0195] In aspect 42, the wearable display system of aspect 41, wherein, in order to authenticate a companion device, the processor is programmed to verify the consistency between the decrypted public key and the public key of the companion device.

[0196] In aspect 43, a wearable display system of any of aspects 34-42, wherein the first communication channel is a wireless communication channel.

[0197] In aspect 44, the wearable display system of aspect 43, wherein the wireless communication channel includes a Wi-Fi communication channel or a near field communication (NFC) channel.

[0198] In aspect 45, the wearable display system of any of aspects 34-44, wherein the companion device includes at least one of another wearable display system, head-mounted display system, totem, mobile phone, tablet computer, mobile device or any combination thereof.

[0199] In aspect 46, a wearable display system of any of aspects 34-45, wherein the processor is further programmed by executable instructions to: notify the user companion device of the wearable display system that it has been authenticated.

[0200] In aspect 47, the wearable display system of aspect 46, wherein, in order to notify the user of the wearable display system, the processor is programmed to: use a display to notify the user of the wearable display system that a companion device has been authenticated.

[0201] In aspect 48, in any of aspects 46-47, a wearable display system wherein, in order to notify the user of the wearable display system, the processor is programmed to: cause the display to modify the user interface shown to the user to notify the user of the wearable display system that the companion device has been authenticated.

[0202] In aspect 49, a wearable display system of any of aspects 46-48, wherein, in order to notify the user of the wearable display system, the processor is programmed to use the speaker of the wearable display system to notify the user that a companion device has been authenticated.

[0203] In aspect 50, a method for device authentication is disclosed. The method, under the control of a hardware processor, includes: receiving a data object via a communication channel; receiving an image of an optical pattern displayed by a companion device, wherein the companion device generates the optical pattern using shared data; extracting first data from the optical pattern in the received image; generating a transformation of the data object using the first data; and sending the transformation of the data object to the companion device via the communication channel for authentication.

[0204] In aspect 51, the method of aspect 50, wherein the data object includes challenge text.

[0205] In aspect 52, the method of any of aspects 50-51, wherein the transformation of the data object includes hashing the data object generated using the first data.

[0206] In aspect 53, the method of any of aspects 50-52, wherein the transformation of the data object includes an encrypted data object generated using the first data.

[0207] In aspect 54, the method of any of aspects 50-53, wherein the first data includes shared data.

[0208] In aspect 55, the method of any one of aspects 50-54, wherein for authentication, the companion device is configured to: generate a transformation of a data object using shared data; receive the transformation of the data object; and verify the consistency between the received transformation of the data object and the generated transformation of the data object.

[0209] In aspect 56, a head-mounted display system is disclosed. The head-mounted display system includes: a display; a non-transitory computer-readable storage medium storing executable instructions; and a processor communicating with the non-transitory computer-readable storage medium, the processor being programmed by the executable instructions to: receive first communication from a device, wherein the first communication is generated by the device based on first shared information; extract first information from the first communication; and authenticate the device based on the first information extracted from the first communication.

[0210] In aspect 57, the head-mounted display system of aspect 56, wherein the first communication includes optical communication.

[0211] In aspect 58, the head-mounted display system of aspect 57, wherein optical communication includes a one-dimensional pattern, a two-dimensional optical pattern, or a combination thereof.

[0212] In aspect 59, a head-mounted display system of any of aspects 56-58, wherein the first communication includes audio communication.

[0213] In aspect 60, aspect 59 describes a head-mounted display system in which audio communication is inaudible to the human ear.

[0214] In aspect 61, the head-mounted display system of aspect 59, wherein audio communication is ultrasonic.

[0215] In aspect 62, a head-mounted display system of any of aspects 56-61, wherein the first shared information includes spoken phrases.

[0216] In aspect 63, the head-mounted display system of aspect 62 further includes a microphone configured to capture the spoken phrase, wherein the processor is further programmed to receive the spoken phrase.

[0217] In aspect 64, aspect 63 includes a head-mounted display system in which the device’s microphone is configured to receive the spoken phrase.

[0218] In aspect 65, a head-mounted display system of any of aspects 62-64, wherein the spoken phrases include alphanumeric phrases.

[0219] In aspect 66, a head-mounted display system in any of aspects 56-65, wherein the first shared information includes one or more spectral characteristics of spoken phrases.

[0220] In aspect 67, a head-mounted display system of any of aspects 56-66, wherein the first shared information includes the user's biometric information.

[0221] In aspect 68, aspect 67 includes a head-mounted display system, wherein the processor is further programmed to extract the user's biometric information.

[0222] In aspect 69, a head-mounted display system of any of aspects 56-68, wherein the processor is further programmed to: generate first shared information; and transmit the first shared information to a device via a first communication channel.

[0223] In aspect 70, a head-mounted display system of any of aspects 56-69, wherein, for the purpose of authenticating the device, the processor is programmed to verify the consistency of first information and first shared information.

[0224] In aspect 71, a head-mounted display system of any of aspects 56-70, wherein the processor is further programmed to: receive second communication from a device, wherein the second communication is generated by the device based on second shared information, and wherein the second shared information is generated by the device; extract second information from the received second communication; and transmit the second shared information to the device via a first communication channel.

[0225] In aspect 72, aspect 71 includes a head-mounted display system, wherein the device is programmed to: receive second information from the head-mounted display system; and authenticate the head-mounted display system based on the received second information.

[0226] In aspect 73, the head-mounted display system of aspect 72, wherein, in order to verify the head-mounted display system, the device is programmed to: verify the consistency between the received second information and the generated second shared information.

[0227] In aspect 74, a head-mounted display system of any of aspects 56-73, wherein the first shared information is generated by the device.

[0228] In aspect 75, a head-mounted display system of any of aspects 56-74, wherein the processor is further programmed to: receive a public key of a device; receive an encrypted public key of the device via a first communication channel, wherein the encrypted public key of the device is encrypted by the device using first shared information; and decrypt the encrypted public key of the device using the first shared information to obtain a decrypted public key.

[0229] In aspect 76, the head-mounted display system of aspect 75, wherein, in order to verify the device, the processor is programmed to verify the consistency between the decrypted public key and the device's public key.

[0230] In aspect 77, a head-mounted display system of any of aspects 56-76, wherein the first communication channel is a wireless communication channel.

[0231] In aspect 78, the head-mounted display system of aspect 77, wherein the wireless communication channel includes a Wi-Fi communication channel, a near-field communication (NFC) channel, or a combination thereof.

[0232] In aspect 79, a head-mounted display system of any of aspects 56-78, wherein the device includes at least one of another head-mounted display system, a head-mounted display system, a totem, a mobile phone, a tablet computer, a mobile device, or any combination thereof.

[0233] In aspect 80, a head-mounted display system according to any of aspects 56-79, wherein the processor is further programmed to notify the user equipment of the head-mounted display system that it has been authenticated.

[0234] In aspect 81, aspect 80 includes a head-mounted display system, wherein, in order to notify the user of the head-mounted display system, the processor is programmed to use a display to notify the user of the head-mounted display system that the device has been authenticated.

[0235] In aspect 82, in any of aspects 80-81, a head-mounted display system, wherein, in order to notify a user of the head-mounted display system, the processor is programmed to: cause the display to modify the user interface shown to the user to notify the user device of the head-mounted display system that it has been authenticated.

[0236] In aspect 83, a head-mounted display system of any of aspects 80-82, wherein, in order to notify the user of the head-mounted display system, the processor is programmed to use the speaker of the head-mounted display system to notify the user of the head-mounted display system that a companion device has been authenticated.

[0237] in conclusion

[0238] Each process, method, and algorithm described herein and / or in the accompanying drawings may be embodied in a code module executed by one or more physical computing systems, hardware computer processors, application-specific circuitry, and may be fully or partially automated, and / or embodied in electronic hardware configured to execute specific computer instructions. For example, a computing system may include a general-purpose computer (e.g., a server) or a special-purpose computer programmed with specific computer instructions, special-purpose circuitry, etc. The code module may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some implementations, specific operations and methods may be performed by circuitry specific to a given function.

[0239] Furthermore, some implementations of the functions disclosed herein are mathematically, computationally, or technically complex enough that they may require application-specific hardware or one or more physical computing devices (using appropriate special-purpose executable instructions) to perform the function, for example, due to the amount or complexity of the computations involved, or the need to provide results substantially in real time. For example, video may comprise many frames, each with millions of pixels, and requires specially programmed computer hardware to process the video data in order to provide the required image processing tasks or applications within a commercially reasonable timeframe.

[0240] Code modules or any type of data can be stored on any type of non-transitory computer-readable medium, including hard disk drives, solid-state storage, random access memory (RAM), read-only memory (ROM), optical disks, volatile or non-volatile memory, and physical computer memory of the same and / or similar combinations. Methods and modules (or data) can also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) over various computer-readable transmission media, including wireless-based and wire / cable-based media, and can take many forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps can be persistently or otherwise stored in any type of non-transitory, tangible computer storage, or can be transmitted via computer-readable transmission media.

[0241] Any process, block, state, step, or function described herein and / or in the flowcharts depicted in the accompanying drawings should be understood to potentially represent code modules, segments, or portions of code, including one or more executable instructions (e.g., logical or arithmetic instructions) for implementing a particular function or step within a process. Various processes, blocks, states, steps, or functions can be combined, rearranged, added, deleted, modified, or otherwise altered from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functions described herein. The methods and processes described herein are not limited to any particular order, and associated blocks, steps, or states may be executed in other suitable orders, such as serial, parallel, or otherwise. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described herein is for illustrative purposes and should not be construed as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems can generally be integrated into a single computer product or packaged into multiple computer products. Many implementation variations are possible.

[0242] Processes, methods, and systems can be implemented in a networked (or distributed) computing environment. Networked environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network can be wired or wireless, or any other type of communication network.

[0243] The systems and methods disclosed herein each have several inventive aspects, none of which individually assumes responsibility for or claims the desired properties disclosed herein. The various features and processes described herein can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Various modifications to the implementations described herein will be apparent to those skilled in the art, and the general principles defined herein can be applied to other implementations without departing from the spirit or scope of this disclosure. Therefore, the claims are not limited to the implementations shown herein but should be given the widest scope consistent with this disclosure, the principles disclosed herein, and the novel features.

[0244] Some features described in this specification in the context of independent implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually in multiple implementations or in any suitable sub-combination. Furthermore, although the features described above may be described as functioning in certain combinations, or even initially claimed to be so, in some cases, one or more features from the claimed combination may be removed from the claimed combination, and the claimed combination may refer to a sub-combination or a variation of the sub-combination. For each and every embodiment, no single feature or group of features is necessary or indispensable.

[0245] Conditional language used herein, such as “capable,” “can,” “may,” “can,” “e.g.,” etc., unless specifically stated otherwise or otherwise understood in the context in which they are used, is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while others do not, and that certain features, elements, and / or steps are not included. Therefore, such conditional language is generally not intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments necessarily include logic for determining, with or without author input or prompting, whether such features, elements, and / or steps are included or performed in any particular embodiment. The terms “comprising,” “including,” “having,” etc., are synonyms and are used inclusively in an open-ended manner and do not exclude additional elements, features, actions, operations, etc. Furthermore, the term “or” is used in its inclusive sense (rather than in its exclusive sense), and thus, when used, for example, to connect lists of elements, the term “or” means one, some, or all of the elements in the list. Moreover, unless otherwise specified, the terms “a,” “an,” and “the” as used in this application and the appended claims shall be construed as “one or more” or “at least one.”

[0246] As used herein, the phrase “at least one” in a list of items refers to any combination of these items, including individual members. For example, “at least one” in “A, B, or C” is intended to cover: A, B, C, A and B, A and C, B and C, and A, B, and C. Unless otherwise specifically stated, connective language such as the phrase “at least one of X, Y, and Z” is otherwise understood in the context of common use to express that an item, term, etc., is at least one of X, Y, or Z. Therefore, such connective language generally does not imply that some embodiments require the presence of at least one of X, at least one of Y, and at least one of Z.

[0247] Similarly, while operations may be described in a specific order in the accompanying drawings, it should be understood that it is not necessary to perform these operations in the specific order or sequence shown, or to perform all of the shown operations, to obtain the desired result. Furthermore, the drawings may schematically depict more than one example process in the form of a flowchart. However, other operations not described may be incorporated into the example methods and processes schematically shown. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the shown operations. Additionally, in other implementations, operations may be rearranged or reordered. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above implementations should not be construed as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products. Additionally, other implementations are also within the scope of the following claims. In some cases, the actions described in the claims may be performed in a different order, and the desired result may still be obtained.

Claims

1. A head-mounted display system, comprising: Non-transitory computer-readable storage medium that stores executable instructions; as well as One or more processors communicating with the non-transitory computer-readable storage medium, the one or more processors being programmed by the executable instructions to: A request is received from a first device to authenticate a first connection between the first device and the head-mounted display system, wherein the request includes first data, the first data being a number encoded in a blinking pattern by two or more light-emitting devices on the first device, the two or more light-emitting devices including a first light-emitting device configured to emit light of a first color and a second light-emitting device configured to emit light of a second color, wherein the number is encoded in the blinking pattern according to a predetermined protocol such that a first digit of the number corresponds to the number of blinks of the first light-emitting device of the first color and a second digit of the number corresponds to the number of blinks of the second light-emitting device of the second color; Extract the first data from the flashing pattern; The first data is sent to an authentication device, which is configured to authenticate the first connection based at least in part on a first comparison of the first shared data received from the first device with the first data. Receive confirmation of successful authentication of the first connection from the authentication device, the confirmation being at least in part based on determining a match between the first shared data and the first data; as well as In response to receiving the confirmation of successful authentication of the first connection, Allowing bidirectional data communication between the first device and the head-mounted display system, and The user of the head-mounted display system is notified that the first connection has been authenticated.

2. The head-mounted display system according to claim 1, wherein, The first data is associated with the first shared data.

3. The head-mounted display system according to claim 1, wherein, The first device is configured to send the first shared data to the authentication device.

4. The head-mounted display system according to claim 1, wherein, The first comparison is based at least in part on the verification of consistency between the first shared data and the first data.

5. The head-mounted display system according to claim 1, wherein, The first device and the authentication device are different devices.

6. The head-mounted display system according to claim 1, wherein, The bidirectional communication of data between the first device and the head-mounted display system is performed through a communication channel.

7. The head-mounted display system according to claim 1, wherein, The two or more light-emitting devices include one or more light-emitting diodes.

8. The head-mounted display system according to claim 1, wherein, The authentication device communicates with the head-mounted display system and the first device via the cloud.

9. The head-mounted display system of claim 1, wherein the one or more processors are further programmed by the executable instructions to: The identifier of the head-mounted display system is sent to the authentication device.

10. The head-mounted display system according to claim 9, wherein, The authentication device authenticates the first connection based at least in part on the identifier.

11. The head-mounted display system according to claim 1, wherein, The request was generated by the first device.

12. The head-mounted display system according to claim 1, wherein, The first data is generated by the authentication device based at least in part on the first shared data.

13. The head-mounted display system according to claim 1, wherein, The first shared data includes one or more of the following: the identifier of the first device; spoken phrases; One or more spectral characteristics of spoken phrases; or the user's biometric data.

14. The head-mounted display system according to claim 1, wherein, The first device is a companion device, which is configured to receive user input and provide commands to the head-mounted display system using the first connection based on the user input.

15. A method for device authentication, comprising: A request is received from a first device to authenticate a first connection between the first device and a head-mounted display system, wherein the request includes first data, the first data being a number encoded in a blinking pattern by two or more light-emitting devices on the first device, the two or more light-emitting devices including a first light-emitting device configured to emit light of a first color and a second light-emitting device configured to emit light of a second color, wherein the number is encoded in the blinking pattern according to a predetermined protocol such that a first digit of the number corresponds to the number of blinks of the first light-emitting device of the first color and a second digit of the number corresponds to the number of blinks of the second light-emitting device of the second color; Extract the first data from the flashing pattern; The first data is sent to an authentication device, which is configured to authenticate the first connection based at least in part on a first comparison of the first shared data received from the first device with the first data. Receive confirmation of successful authentication of the first connection from the authentication device, the confirmation being at least in part based on determining a match between the first shared data and the first data; as well as In response to receiving the confirmation of successful authentication of the first connection, Allowing bidirectional data communication between the first device and the head-mounted display system, and The user of the head-mounted display system is notified that the first connection has been authenticated.

16. The method according to claim 15, wherein, The first device is configured to send the first shared data to the authentication device.

17. The method according to claim 15, wherein, The first comparison is based at least in part on the verification of consistency between the first shared data and the first data.

18. The method according to claim 15, wherein, The first device and the authentication device are different devices.

19. The method according to claim 15, wherein, The bidirectional communication of data between the first device and the head-mounted display system is performed through a communication channel.

20. The method of claim 15, wherein, The first device is a companion device, which is configured to receive user input and provide commands to the head-mounted display system using the first connection based on the user input.

21. A display system, comprising: Head-mounted display; Memory that stores executable instructions; as well as One or more processors communicating with the memory and the memory, said one or more processors being programmed by the executable instructions to perform the following operations: Receive a request to authenticate the connection between the display system and the companion device; In response to receiving the request to form the connection, first data is determined to be used to authenticate the connection between the display system and the companion device, wherein the first data is based at least in part on audio data spoken by the current user of the display system; The first data is sent to an authentication device, which is configured to authenticate the connection based at least in part on comparing the first data with second data received by the authentication device from the companion device, and the second data is based at least in part on the audio data; The authentication device receives confirmation of successful authentication of the connection, wherein the successful authentication is determined by the authentication device based at least in part on determining a match between the first data and the second data; as well as In response to receiving confirmation of successful connection authentication, This allows for bidirectional data communication between the companion device and the display system using the connection, and The display presents a notification to the current user of the display system that the connection has been authenticated.

22. The display system according to claim 21, wherein, The request to form a connection between the display system and the companion device is received from the companion device.

23. The display system according to claim 22, wherein, The request includes the first data generated by the companion device based on the audio data, and wherein determining the first data includes extracting the first data from the request.

24. The display system according to claim 23, wherein, The companion device is configured to capture the audio data and generate the first data based on the audio data, and the companion device is further configured to generate the second data based on the audio data and transmit the second data to the authentication device.

25. The display system according to claim 22, wherein, The request to form the connection between the display system and the companion device is received from the companion device via out-of-band communication.

26. The display system of claim 21, further comprising at least one microphone, wherein, Determining the first data includes: The audio data is captured using the at least one microphone; and The first data is generated based at least in part on the captured audio data.

27. The display system according to claim 26, wherein, The companion device is configured to capture the audio data, generate the second data based on the audio data, and transmit the second data to the authentication device.

28. The display system according to claim 21, wherein, At least one of the first data or the second data includes one or more spectral characteristics of the audio data.

29. The display system according to claim 21, wherein, The companion device is configured to receive user input from the user and, based on the user input, use the connection to provide commands to control the display system.

30. The display system according to claim 21, wherein, The companion device is a portable computing device.

31. A method for authenticating a connection between a companion device and a display system including a head-mounted display, comprising: Receive a request to authenticate the connection between the display system and the companion device; In response to receiving the request to form the connection, first data is determined to be used to authenticate the connection between the display system and the companion device, wherein the first data is based at least in part on audio data spoken by the current user of the display system; The first data is sent to an authentication device, which is configured to authenticate the connection based at least in part on comparing the first data with second data received by the authentication device from the companion device, and the second data is based at least in part on the audio data; The authentication device receives confirmation of successful authentication of the connection, wherein the successful authentication is determined by the authentication device based at least in part on determining a match between the first data and the second data; as well as In response to receiving the confirmation of successful authentication of the connection, This allows for bidirectional data communication between the companion device and the display system using the connection, and The display presents a notification to the current user of the display system that the connection has been authenticated.

32. The method according to claim 31, wherein, The request to form a connection between the display system and the companion device is received from the companion device.

33. The method according to claim 32, wherein, The request includes the first data generated by the companion device based on the audio data, and wherein determining the first data includes extracting the first data from the request.

34. The method according to claim 33, wherein, The companion device is configured to capture the audio data and generate the first data based on the audio data, and the companion device is further configured to generate the second data based on the audio data and transmit the second data to the authentication device.

35. The method according to claim 32, wherein, The request to form the connection between the display system and the companion device is received from the companion device via out-of-band communication.

36. The method of claim 31, further comprising at least one microphone, wherein, Determining the first data includes: The audio data is captured using the at least one microphone; and The first data is generated based at least in part on the captured audio data.

37. The method of claim 36, wherein, The companion device is configured to capture the audio data, generate the second data based on the audio data, and transmit the second data to the authentication device.

38. The method according to claim 31, wherein, At least one of the first data or the second data includes one or more spectral characteristics of the audio data.

39. The method according to claim 31, wherein, The companion device is configured to receive user input from the user and, based on the user input, use the connection to provide commands to control the display system.

40. The method according to claim 31, wherein, The companion device is a portable computing device.

Images