Head mounted display system configured to exchange biometric information

Abstract

A head mounted display system is disclosed that is configured to facilitate the exchange of biometric information between the head mounted display system and another computing device. The head mounted display system may include a virtual or augmented reality device. After displaying a consent request regarding the biometric information with the head mounted display system, a response to the consent request including an indication of consent regarding aspects of the biometric information may be determined. After obtaining biometric information from a wearer using, for example, a camera of the head mounted display and processing the biometric information, a biometric information processing result may be generated. The result may be transmitted from the head mounted display system to another computing device.

Description

[0001] This application is a divisional application of the Chinese patent application with application number 201780031396.8. The application date of the original application is March 17, 2017, the priority date is March 22, 2016, the date of entering the Chinese national phase is November 21, 2018, and the name of the invention is "Head-mounted display system configured to exchange biometric information".

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority to U.S. Patent Application Serial No. 62 / 311,745, filed on March 22, 2016, entitled “BIOMETRIC INFORMATION EXCHANGE SYSTEM,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0004] The present disclosure relates to virtual reality and augmented reality imaging and visualization systems, and more particularly to systems and methods for exchanging information between such imaging and visualization systems. Background Art

[0005] Wearable display systems such as virtual or augmented reality devices can capture biometric information of users. Biometric information can include physiological characteristics and / or behavioral characteristics associated with the wearer. Physiological characteristics can include body posture (e.g., eye gaze), pupil dilation state, facial features (e.g., grin or frown), voice characteristics, pulse rate or blood pressure, skin condition (e.g., sweating), etc. Behavioral characteristics can include the wearer's gait, voice characteristics, tendencies, etc. Summary of the invention

[0006] Disclosed herein are embodiments of a wearable display system, such as an augmented or virtual reality display device, configured to exchange biometric information of a wearer of the wearable display system. In some embodiments, a head-mounted display system includes: a first image capture device configured to capture multiple eye images of a user's eyes; a second image capture device configured to capture multiple external world images; a display configured to display virtual images to the eyes of a wearer of the head-mounted display; and a hardware processor that communicates with the first capture device, the second capture device, and the display, the hardware processor being programmed to: use the display to transmit a consent request to a wearer of the head-mounted display, the request including multiple consent categories for biometric information of the wearer; receive a response to the consent request; determine that the response includes a consent indication from the wearer of the head-mounted display, the consent indication being associated with a consent category that the wearer has indicated as agreeing to; obtain multiple eye images of the wearer of the head-mounted display captured by the first image capture device; and in response to a request for biometric information from a biometric information acquirer computing device, transmit the multiple eye images to the biometric information acquirer computing device based at least in part on the consent categories that the wearer has indicated as agreeing to and the request for biometric information.

[0007] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither the present summary nor the following detailed description is intended to define or limit the scope of the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An illustration of an augmented reality scene with certain virtual reality objects and certain real reality objects viewed by a person is depicted.

[0009] Figure 2 An example of a wearable display system is schematically shown.

[0010] Figure 3 Aspects of a method of simulating a three-dimensional image using multiple depth planes are schematically illustrated.

[0011] Figure 4 An example of a waveguide stack for outputting image information to a user is schematically shown.

[0012] Figure 5 An example output light beam that may be output by a waveguide is shown.

[0013] Figure 6is a schematic diagram showing a display system that includes a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem, and is used for the generation of a multi-focal volumetric display, image or light field.

[0014] Figure 7 is a block diagram depicting an illustrative operating environment in which a biometric information exchange system exchanges information between a wearable display system and a computing device that is a potential acquirer of a wearer's biometric information.

[0015] Fig. 8A An example display page generated by a biometric information exchange system for a wearer of a wearable display system to provide consent to view a biometric information offer is shown.

[0016] Figure 8B An example display page generated by the biometric information exchange system is shown, and when the wearer agrees to view Fig. 8A The quote provided by the initiated biometric information exchange system may be a corresponding pop-up window displayed on the example display page.

[0017] Figure 8C An example display page generated by a biometric information exchange system for a wearer to provide consent to receive an offer for certain categories of biometric information about the wearer is shown.

[0018] Fig.8D An example display page generated by the biometric information exchange system and when the wearer agrees to use the biometric information exchange system to receive Figure 8C The initiated offer for certain categories of biometric information may be a corresponding pop-up window displayed on the example display page.

[0019] Fig. 8E An example display page generated by a biometric information exchange system for a wearer to provide consent to share the wearer's biometric information is shown.

[0020] Fig. 9 A flow chart of an illustrative method for exchanging biometric information is shown.

[0021] Throughout the drawings, reference numbers may be used again to indicate correspondence between reference elements.The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION

[0022] Sample augmented reality scene

[0023] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears real or can be perceived as real. Virtual reality "VR" scenarios typically involve presenting digital or virtual image information in a manner that is opaque to other actual real-world visual input; augmented reality "AR" scenarios typically involve presenting digital or virtual image information as an enhancement to the actual world visualization around the user; or mixed reality "MR" scenarios, which typically involve merging the real world and the virtual world to produce a new environment in which physical and virtual objects coexist and interact in real time. It turns out that the human visual perception system is very complex, and it is a challenge to produce VR, AR, or MR technologies that promote a comfortable, natural-feeling, rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies.

[0024] Figure 1 An illustration of an augmented reality scene with some virtual reality objects and some real reality objects viewed by a person is shown. Figure 1 An augmented reality scene 100 is depicted in which a user of AR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the AR technology also perceives that he "sees" a robotic statue 130 standing on the real-world platform 120, and a flying cartoon-like avatar character 140 (e.g., Bumblebee) that appears to be an avatar of Bumblebee, even though these elements do not exist in the real world.

[0025] In order for a three-dimensional (3-D) display to produce a realistic sense of depth, and more specifically, a simulated sense of surface depth, it is desirable for each point in the display's field of view to generate an accommodation response that corresponds to its virtual depth. If the accommodation response to a displayed point does not correspond to the virtual depth of that point, as determined by the binocular depth cues of convergence and stereoscopic vision, the human eye may experience an accommodation conflict, resulting in unstable imaging, detrimental eye strain, headaches, and, in the absence of accommodation information, an almost complete lack of surface depth.

[0026] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The image may be different for each depth plane (e.g., providing a slightly different presentation of a scene or object) and may be focused on separately by the viewer's eyes, thereby helping to provide depth cues to the user based on the eye adaptation required to focus different image features of a scene located at different depth planes and / or based on observing different image features at different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a credible sense of depth. In order to generate or enhance VR, AR, and MR experiences, the display system may use biometric information to enhance these experiences. For example, as described herein, a biometric information exchange system may facilitate the exchange of biometric information between a wearer of a mixed reality display system and a biometric information acquirer computing device.

[0027] A mixed reality device (MRD), such as a head-mounted MR display system, can capture a wealth of biometric information about the MRD wearer. The biometric information may include, for example, information about the wearer's head or eye movements (e.g., where the user is looking), pupil size (e.g., which may indicate the wearer's interest or excitement), timing (e.g., how long the user continues to look at an item), imagery of the wearer's environment (e.g., what the wearer is looking at), the wearer's location, and the like. The MRD may include sensors to measure the wearer's heart rate, pulse, blood pressure, skin reactions (e.g., sweating), brain activity, and the like. This biometric information may be stored in the device or in a network storage system and may include a dataset of extensive detailed information about the MRD wearer. Although described above in the context of a mixed reality device, any wearable display system (e.g., Figure 2 The wearable display system 200 or Figure 4 and Figure 6 The display system 400 in can be used with the systems and methods described herein.

[0028] An entity may wish to obtain some portion of the wearer's biometric information. For example, a large retailer may wish to determine when the wearer visits its retail store and for how long, where the wearer moves within the store, what items the wearer looks at and for how long, etc. As another example, a game developer may wish to determine what game characters the wearer looks at during game play, what choices the wearer makes when interacting with the characters, the wearer's interest, fear, or excitement level in playing the game, etc.

[0029] Wearers of wearable display systems may be concerned about the confidentiality of their respective biometric information. Similarly, potential acquirers of biometric information may wish to keep such biometric information absolutely confidential. Therefore, both the wearer and the potential acquirer of biometric information may wish to utilize systems and methods for facilitating the exchange of biometric information that maintain the confidentiality of biometric information. The biometric information exchange system described herein maintains the confidentiality of biometric information. For example, as further described below, before accessing any biometric information associated with the wearable display system, the biometric information exchange system may transmit a consent request for biometric information to the wearer of the wearable display system. If the biometric information exchange system does not receive an indication of consent from the wearer, biometric information cannot be exchanged on the exchange system. On the other hand, if the wearer indicates an agreement to the consent request, the biometric information exchange system may facilitate access to the wearer's biometric information, for example, via a secure connection.

[0030] Generally speaking, once the wearer has indicated assent to a request to indicate consent, the wearable display system can utilize a biometric information exchange system to provide biometric information to a computing device of a biometric information acquirer, e.g., the biometric information acquirer computing device can obtain biometric information retrieved from the wearable display system via a secure connection during the biometric information exchange. If the biometric information is maintained with absolute confidentiality, the exchange system can authorize access to the wearer's biometric information via the secure connection, either via the secure connection where the information is transmitted to the computing device of the biometric information acquirer, or via an exchange system that facilitates storage of such biometric information for future exchange. Thus, the systems and methods disclosed herein facilitate information exchange between a wearable display system and a computing device of a potential acquirer of biometric information.

[0031] Exemplary Wearable Display System

[0032] Figure 2An example of a wearable display system 200 is shown that can be used to present a VR, AR, or MR experience to a display system wearer or viewer 204. The wearable display system 200 can be programmed to perform any application or embodiment described herein (e.g., eye image processing). The display system 200 includes a display 208 and various mechanical and electronic modules and systems that support the functionality of the display 208. The display 208 can be coupled to a frame 212 that can be worn by a display system user, wearer, or viewer 204, and the frame is configured to position the display 208 in front of the eyes of the wearer 204. The display 208 can be a light field display. In some embodiments, a speaker 216 is coupled to the frame 212 and positioned near the user's ear canal. In some embodiments, another speaker (not shown) is positioned near the user's other ear canal to provide stereo / shapeable sound control. The display system 200 may include an externally facing imaging system (e.g., Figure 4 The display system 200 may also include an interior-facing imaging system (e.g., Figure 4 The display 208 is operably coupled 220 to a local data processing module 224, such as by a wired lead or a wireless connection, which can be mounted in a variety of configurations, such as fixedly attached to the frame 212, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 204 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0033] The frame 212 may have one or more cameras attached or mounted to the frame 212 to obtain images of one or more eyes of the wearer. In one embodiment, the one or more cameras may be mounted to the frame 212 in front of the eyes of the wearer so that the eyes may be imaged directly. In other embodiments, the cameras may be mounted along the stem of the frame 212 (e.g., near the ears of the wearer). In such embodiments, the display 208 may be coated with a material that reflects light from the wearer's eyes toward the cameras. The light may be infrared light, as iris features stand out in infrared images.

[0034] The local processing and data module 224 may include a hardware processor and non-transitory digital memory, such as non-volatile memory (e.g., flash memory), both of which may be used to assist in the processing, capture, and storage of data. The data may include: (a) data captured from a sensor (e.g., which may be operably coupled to the frame 212 or otherwise attached to the user 204), such as an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a gyroscope, a global positioning satellite (GPS unit, a radio, a voice detector, a potentiometric sensor, a pulse or blood pressure sensor, an electrocardiogram sensor, and / or an electroencephalogram sensor; and / or (b) data acquired and / or processed using the remote processing module 228 and / or the remote data repository 232. The data After such processing or retrieval, it may be transferred to the display 208. The local processing and data module 224 may be operably coupled to the remote processing module 228 and the remote data repository 232 via communication links 236 and / or 240, such as via a wired or wireless communication link, so that these remote modules 228, 232 can be used as resources for the local processing and data module 224. One or more image capture devices may be used to capture eye images used in the eye image processing process. In addition, the remote processing module 228 and the remote data repository 232 may be operably coupled to each other.

[0035] In some embodiments, the remote processing module 228 may include one or more processors configured to analyze and process data and / or image information, such as video information captured by an image capture device. Video data may be stored locally in the local processing and data module 224 and / or in the remote data repository 232. In some embodiments, the remote data repository 232 may include a digital data storage facility that may be available via the Internet or other network configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 224, allowing for fully autonomous use from the remote module.

[0036] In some embodiments, the local processing and data module 224 and / or the remote processing module 228 are programmed to perform embodiments of the systems and methods as described herein. The image capture device can capture video for a specific application (e.g., video of the wearer's eyes for an eye tracking application or video of the wearer's hands or fingers for a gesture recognition application). The video can be analyzed by one or both of the processing modules 224, 228. In some cases, offloading at least some of the iris code generation to a remote processing module (e.g., to the "cloud") can improve the efficiency or speed of the computation. Parameters of the systems and methods disclosed herein can be stored in the data modules 224 and / or 228.

[0037] One or both of the processing modules 224, 228 can use the results of the analysis for additional operations or processing. For example, in various applications, the wearable display system 200 can use biometric recognition, eye tracking, gesture, object, posture recognition or classification, etc. For example, the wearable display system 200 can analyze the captured video of the hands of the wearer 204 and recognize gestures from the wearer's hands (e.g., picking up a real or virtual object, sending a signal of agreement or disagreement (e.g., "thumbs up" or "thumbs down"), etc.), as well as the wearable display system.

[0038] In some implementations, the display system 200 may also include an electrical skin sensor (e.g., a galvanic sensor) that is operable to detect skin sensing activity such as skin conductance or galvanic skin response, which may indicate activity in the wearer's autonomic nervous system (e.g., causing sweating or indicating stress). The display system 200 may also include other types of electrical sensors, such as electrical sensors that detect brain activity (e.g., as performed in an electroencephalogram test) or electrical sensors that detect heart activity (e.g., as performed in an electrocardiogram test). The electrical skin sensor and / or the electrical sensor may be operatively coupled to the display system 200 via a communication link, such as via a wired or wireless communication link. For example, the wired or wireless communication link may couple the electrical skin sensor and / or the electrical sensor to a local processing and data module 224, which in turn is associated with any processing module that is operable to process biometric information of the display system 200. Thus, as an example, the electrical skin sensor may detect electrical skin activity or other measurements associated with a glance of a wearer of the display system 200. Such biometric information may also be processed by the display system 200.

[0039] The human visual system is complex, and providing a realistic sense of depth is challenging. Without being limited by theory, it is believed that a viewer of an object may perceive the object to be three-dimensional due to a combination of vergence and accommodation. The vergence movement of the two eyes relative to each other (e.g., the rotational movement of the pupils toward or away from each other to converge the eyes' line of sight to fixate on an object) is closely related to the focusing (or "accommodation") of the lenses of the eyes. Under normal circumstances, changing the focus of the lenses of the eyes, or adjusting the eyes, so that the focus changes from one object to another at a different distance, will automatically result in a matching change in vergence to the same distance, in a relationship known as the "accommodative vergence reflex." Likewise, under normal circumstances, a change in vergence will trigger a matching change in accommodation. A display system that provides a better match between accommodation and vergence can form a more realistic or more comfortable simulation of a three-dimensional image.

[0040] Figure 3Various aspects of a method for simulating a three-dimensional image using multiple depth planes are shown. Figure 3 , eyes 302 and 304 adapt to objects at various distances from eyes 302 and 304 on the z-axis so that these objects are in focus. Eyes 302 and 304 present a specific adaptation state to focus on objects at different distances along the z-axis. Therefore, it can be said that a specific adaptation state is associated with a specific one of the depth planes 306, with an associated focal length, so that when the eye is in an adaptation state for the depth plane, objects or portions thereof in the specific depth plane are in focus. In some embodiments, a three-dimensional image can be simulated by providing a different presentation of an image for each of eyes 302 and 304, and also by providing a different presentation of an image corresponding to each of the depth planes. Although shown separately for clarity of explanation, it should be understood that, for example, the fields of view of eyes 302 and 304 can overlap as the distance along the z-axis increases. In addition, although shown as a plane for simplicity of explanation, it should be understood that the contour of the depth plane can be curved in physical space so that all features in the depth plane are in focus with an eye in a specific adaptation state. Without being limited by theory, it is believed that the human eye can generally interpret a limited number of depth planes to provide a sense of depth. Therefore, a highly credible simulation of perceived depth can be achieved by providing the eye with a different presentation of an image corresponding to each of these limited number of depth planes.

[0041] Example Waveguide Stack Assembly

[0042] Figure 4 An example of a waveguide stack for outputting image information to a user is shown. The display system 400 includes a stack of waveguides or a stacked waveguide assembly 405 that can be used to provide a three-dimensional perception to the eye 410 or brain using multiple waveguides 420, 422, 424, 426, 428. In some embodiments, the display system 400 can be used with Figure 2 The system 200 corresponds to, and Figure 4 Some parts of the system 200 are schematically shown in more detail. For example, in some embodiments, the waveguide assembly 405 can be integrated into Figure 2 in the display 208 .

[0043] Continue to refer Figure 4 , the waveguide assembly 405 may also include a plurality of features 430, 432, 434, 436 between the waveguides. In some embodiments, the features 430, 432, 434, 436 may be lenses. In some embodiments, the features 430, 432, 434, 436 may not be lenses. Instead, they may be spacers (e.g., cladding and / or structures for forming air gaps).

[0044] The waveguides 420, 422, 424, 426, 428 and / or the plurality of lenses 430, 432, 434, 436 may be configured to send image information to the eye with different levels of wavefront curvature or light divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. An image injection device 440, 442, 444, 446, 448 may be used to inject image information into the waveguides 420, 422, 424, 426, 428, each of which may be configured to distribute incident light across each respective waveguide for output toward the eye 410. Light exits the output surface of the image injection device 440, 442, 444, 446, 448 and is injected into the corresponding input edge of the waveguides 420, 422, 424, 426, 428. In some embodiments, a single beam of light (e.g., a collimated beam) can be injected into each waveguide to output an entire cloned field of collimated beams directed toward the eye 410 at a specific angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0045] In some embodiments, image injection devices 440, 442, 444, 446, 442 are separate displays that each generate image information for injection into a corresponding waveguide 420, 422, 424, 426, 428. In some other embodiments, image injection devices 440, 442, 446, 446, 448 are outputs of a single multiplexed display that may pipe image information to each of image injection devices 440, 442, 444, 446, 448, for example, via one or more light conduits (e.g., fiber optic cables).

[0046] Controller 450 controls the operation of stacked waveguide assembly 405 and image injection devices 440, 442, 444, 446, 448. In some embodiments, controller 450 includes programming (e.g., instructions in a non-transitory computer readable medium) that adjusts the timing and provides image information to waveguides 420, 422, 424, 426, 428. In some embodiments, controller 450 can be a single integrated device, or a distributed system connected by wired or wireless communication channels. In some embodiments, controller 450 can be a processing module 224 or 228 ( Figure 2In some embodiments, the controller can communicate with an interior-facing imaging system 452 (e.g., a digital camera), an exterior-facing imaging system 454 (e.g., a digital camera), and / or a user input device 456. The interior-facing imaging system 452 (e.g., a digital camera) can be used to capture an image of the eye 410, for example to determine the size and / or orientation of the pupil of the eye 410. The exterior-facing imaging system 454 can be used to image a portion of the world 458. A user can input commands to the controller 450 via the user input device 456 to interact with the display system 400.

[0047] The waveguides 420, 422, 424, 426, 428 can be configured to propagate light within each respective waveguide by total internal reflection (TIR). The waveguides 420, 422, 424, 426, 428 can each be planar or have another shape (e.g., curved) with a top major surface and a bottom major surface and an edge extending between these top major surfaces and bottom major surfaces. In the configuration shown, the waveguides 420, 422, 424, 426, 428 can each include light extraction optical elements 460, 462, 464, 466, 468 that are configured to extract light from the waveguide by redirecting the light propagating within each respective waveguide out of the waveguide to output image information to the eye 410. The extracted light can also be referred to as outcoupled light, and the light extraction optical element can also be referred to as an outcoupling optical element. The waveguide outputs the extracted light beam at the location where the light propagating in the waveguide strikes the light redirecting element. The light extraction optical elements (460, 462, 464, 466, 468) can be, for example, reflective and / or diffractive optical features. Although these light extraction optical elements are shown as being disposed on the bottom major surface of the waveguides 420, 422, 424, 426, 428 for ease of description and drawing clarity, in some embodiments, the light extraction optical elements 460, 462, 464, 466, 468 can be disposed on the top major surface and / or the bottom major surface, and / or can be disposed directly in the volume of the waveguides 420, 422, 424, 426, 428. In some embodiments, the light extraction optical elements 460, 462, 464, 466, 468 can be formed in a material layer attached to a transparent substrate to form the waveguides 420, 422, 424, 426, 428. In some other embodiments, the waveguides 420, 422, 424, 426, 428 may be a single piece of material and the light extraction optical elements 460, 462, 464, 466, 468 may be formed on a surface and / or within the piece of material.

[0048] Continue to refer Figure 4As discussed herein, each waveguide 420, 422, 424, 426, 428 is configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 420 closest to the eye can be configured to pass collimated light injected into such waveguide 420 to the eye 410. The collimated light can represent an optical infinity focal plane. The next upstream waveguide 422 can be configured to emit collimated light, which can be transmitted through a first lens 430 (e.g., a negative lens) before reaching the eye 410. The first lens 430 can be configured to produce a slightly convex wavefront curvature so that the eye / brain interprets the light from the next upstream waveguide 422 as light from the first focal plane that is closer to the eye 410 from optical infinity inward. Similarly, the third upstream waveguide 424 transmits its output light through the first lens 430 and the second lens 432 before reaching the eye 410. The combined optical power of the first lens 430 and the second lens 432 can be configured to produce another wavefront curvature increment so that the eye / brain interprets light from the third waveguide 424 as coming from a second focal plane further inward from optical infinity and closer to the person, rather than from the next upstream waveguide 422.

[0049] The other waveguide layers (e.g., waveguides 426, 428) and lenses (e.g., lenses 434, 436) are similarly configured, and the highest waveguide 428 in the stack sends its output through all lenses between it and the eye to obtain a total optical power that represents the focal plane closest to the person. In order to compensate for the stack of lenses 430, 432, 434, 436 when viewing / interpreting light from the world 458 on the other side of the stacked waveguide assembly 405, a compensating lens layer 438 can be provided at the top of the stack to compensate for the total optical power of the lens stack 430, 432, 434, 436 below. This configuration provides as many perceived focal planes as there are available waveguide / lens pairs. The focusing aspects of the light extraction optical elements 460, 462, 464, 466, 468 of the waveguides 420, 422, 424, 426, 428 and the lenses 430, 432, 434, 436 can be static (e.g., not dynamic or electro-active). In some alternative embodiments, one or both of the outcoupling optics of the waveguide and the focusing aspects of the lens may be dynamic using electrically activated features.

[0050] Continue to refer Figure 4, the light extraction optical elements 460, 462, 464, 466, 468 can be configured to both redirect light out of their corresponding waveguides and output light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. Therefore, waveguides with different associated depth planes can have light extraction optical elements with different configurations that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 460, 462, 464, 466, 468 can be volume features or surface features that can be configured to output light at a specific angle. For example, the light extraction optical elements 460, 462, 464, 466, 468 can be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements, such as diffraction gratings, are described in U.S. Patent Publication 2015 / 0178939, published on June 25, 2015, which is incorporated herein by reference in its entirety. In some embodiments, features 430, 432, 434, 436, 438 may not be lenses. Instead, they may simply be spacers (e.g., cladding and / or structures for forming air gaps).

[0051] In some embodiments, the light extraction optical elements 460, 462, 464, 466, 468 are diffractive features, or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOE has a relatively low diffraction efficiency, so that only a portion of the light beam is deflected toward the eye 410 by each intersection of the DOE, while the remainder continues through the waveguide via total internal reflection. Thus, the light carrying the image information is split into a plurality of related exit beams that exit the waveguide at a plurality of locations, and for a particular collimated beam bouncing within the waveguide, the result is a fairly uniform pattern that exits toward the eye 410.

[0052] In some embodiments, one or more DOEs can be switched between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE can include a polymer dispersed liquid crystal layer in which a droplet includes a diffraction pattern in a host medium, and the refractive index of the droplet can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the droplet can be switched to a refractive index that mismatches the refractive index of the host medium (in which case the pattern actively diffracts incident light).

[0053] In some embodiments, the number and distribution of depth planes and / or depth of field can dynamically change based on the size and / or orientation of the pupil of the viewer's eye. In some embodiments, an inward-facing imaging system 452 (e.g., a digital camera) can be used to capture an image of the eye 410 to determine the size and / or orientation of the pupil of the eye 410. The inward-facing imaging system 452 can be used to obtain images for determining the direction in which the wearer 204 is looking (e.g., eye pose). In some embodiments, the inward-facing imaging system 452 can be attached to the frame 212 (e.g., Figure 2 ), and can be in electrical communication with processing modules 224 and / or 228, which can process image information from the inward-facing imaging system 452 to determine, for example, the pupil diameter of the user 204, or the eye orientation or eye posture.

[0054] In some embodiments, the internal imaging system 452 (e.g., a digital camera) can observe the user's movements, such as eye movements and facial movements. The internal imaging system 452 can be used to capture images of the eye 410 to determine the size and / or orientation of the pupil of the eye 410. The internal imaging system 452 can be used to obtain images for determining the direction in which the user is looking (e.g., eye gestures) or for biometric identification of the user (e.g., via iris recognition). The images obtained by the internal imaging system 452 can be analyzed to determine the user's eye gestures and / or emotions, and the display system 400 can use the gestures and / or emotions to decide which audio or video content should be presented to the user. The display system 400 can also use sensors such as inertial measurement units (IMUs), accelerometers, gyroscopes, etc. to determine head gestures (e.g., head position or head orientation). Head gestures can be used alone or in combination with eye gestures to interact with the main track and / or present audio content.

[0055] In some embodiments, one camera may be used for each eye to determine pupil size and / or orientation of each eye separately, thereby allowing dynamically customized image information to be presented to each eye. In some embodiments, at least one camera may be used for each eye to determine pupil size and / or eye pose separately and independently, thereby allowing dynamically customized image information to be presented to each eye. In some other embodiments, only the pupil diameter and / or orientation of a single eye 410 is determined (e.g., only one camera is used per pair of eyes), and it is assumed that the two eyes of the viewer 204 are similar.

[0056] For example, the depth of field may vary inversely with the size of the viewer's pupil. Thus, as the size of the viewer's eye pupil decreases, the depth of field increases, making one plane indistinguishable because a plane located beyond the eye's depth of focus may become discernible and appear more in focus as the pupil size decreases and the depth of field increases accordingly. Likewise, the number of spaced depth planes used to present different images to the viewer may decrease as the pupil size decreases. For example, without adjusting the eye's accommodation away from one depth plane and to another, a viewer may not be able to clearly perceive details in both the first depth plane and the second depth plane at one pupil size. However, without changing the accommodation, both depth planes may be simultaneously in full focus on the user at another pupil size.

[0057] In some embodiments, the display system can change the number of waveguides that receive image information based on a determination of pupil size and / or orientation, or based on receiving an electrical signal indicative of a particular pupil size and / or orientation. For example, if the user's eye cannot distinguish between two depth planes associated with two waveguides, the controller 450 can be configured or programmed to stop providing image information to one of the waveguides. Advantageously, this can reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for the waveguide is switchable between an on and off state, the DOE can be switched to an off state when the waveguide does receive image information.

[0058] In some embodiments, it may be desirable for the exit beam to satisfy the condition of having a diameter less than the diameter of the viewer's eye. However, given the variation in viewer pupil size, satisfying this condition may be extremely challenging. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the viewer's pupil size. For example, as the pupil size decreases, the size of the exit beam may also decrease. In some embodiments, a variable aperture may be used to vary the exit beam size.

[0059] The display system 400 may include an outward-facing imaging system 454 (e.g., a digital camera) that images a portion of the world 458. This portion of the world 458 may be referred to as a field of view (FOV), and the imaging system 454 is sometimes referred to as a FOV camera. The entire area available for viewing or imaging by the viewer 204 may be referred to as a field of view (FOR). The FOR may include a solid angle of 4π radians around the display system 400. In some implementations of the display system 400, the FOR may include substantially all of the solid angle around the user 204 of the display system 400, because the user 204 may move their head and eyes to see objects around the user (in front of, behind, above, below, or to the side of the user). The images obtained from the outward-facing imaging system 454 may be used to track gestures (e.g., hand or finger gestures) made by the user, detect objects in the world 458 in front of the user, and the like.

[0060] The display system 400 may include a user input device 456 through which a user can input commands to the controller 450 to interact with the display system 400. For example, the user input device 456 may include a touch pad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a stick, a haptic device, a totem (e.g., used as a virtual user input device), etc. In some cases, a user may use a finger (e.g., a thumb) to press or slide on a touch-sensitive input device to provide input to the display system 400 (e.g., to provide user input to a user interface provided by the display system 400). The user input device 456 may be held by a user during use of the display system 400. The user input device 456 may communicate with the display system 400 in wired or wireless communication.

[0061] Figure 5An example of an exit beam output by a waveguide is shown. One waveguide is shown, but it should be understood that other waveguides in the waveguide assembly 405 may function similarly, where the waveguide assembly 405 includes multiple waveguides. Light 505 is injected into the waveguide 420 at the input edge 510 of the waveguide 420 and propagates within the waveguide 420 by total internal reflection (TIR). At the point where the light 505 impinges on the diffractive optical element (DOE) 460, a portion of the light is emitted from the waveguide as an exit beam 515. The exit beams 515 are shown as being substantially parallel, but they may also be redirected to propagate to the eye 410 at an angle (e.g., forming diverging exit beams) that depends on the depth plane associated with the waveguide 420. It should be understood that substantially parallel exit beams may be indicative of a waveguide having light extraction optics that couple out light to form an image that appears to be disposed on a depth plane at a relatively large distance (e.g., optical infinity) from the eye 410. Other waveguides or other light extraction optical element groups may output a more divergent exit beam pattern, which would require the eye 410 to adapt to a closer distance in order to focus it on the retina, and would be interpreted by the brain as light coming from a distance closer to the eye 410 than optical infinity.

[0062] Figure 6 Another example of a display system 400 is shown, which includes a waveguide device, a control subsystem for optically coupling light to or from the waveguide device and the control subsystem. The display system 400 can be used to generate a multi-focal volume, image, or light field. The display system 400 can include one or more main planar waveguides 604 ( Figure 6 The planar waveguides 604 may be similar to those of FIG. Figure 4 The optical system may employ a distributed waveguide arrangement to provide a plurality of waveguides along a first axis (from Figure 6 The distributed waveguide device may, for example, include a distributed planar waveguide 612 and at least one DOE 616 (shown by a double-dashed line) associated with the distributed planar waveguide 612. The distributed planar waveguide 612 may be similar or identical in at least some aspects to the main planar waveguide 604 having a different orientation. Likewise, the at least one DOE 616 may be similar or identical in at least some aspects to the DOE 608. For example, the distributed planar waveguide 612 and / or the DOE 616 may include the same material as the main planar waveguide 604 and / or the DOE 608, respectively. Figure 6 The optical system shown can be integrated into Figure 2In the wearable display system 200 shown.

[0063] The relayed light and exit pupil expanded light are optically coupled from the distributed waveguide device into one or more main planar waveguides 604. The main planar waveguides 604 are arranged along a second axis (e.g., Figure 6 The main planar waveguide 604 can extend the effective exit path of the light along the second axis (e.g., the X-axis). For example, the distribution planar waveguide 612 can relay and extend the light along the longitudinal axis or the Y-axis and pass the light to the main planar waveguide 604, which relays and extends the light along the transverse axis or the X-axis. It is worth noting that the second axis can be an axis that is not orthogonal to the first axis. The main planar waveguide 604 extends the effective exit path of the light along the second axis (e.g., the X-axis). For example, the distribution planar waveguide 612 can relay and extend the light along the longitudinal axis or the Y-axis and pass the light to the main planar waveguide 604, which relays and extends the light along the transverse axis or the X-axis.

[0064] The display system 400 may include one or more sources 620 of colored light (e.g., red, green, and blue lasers) that may be optically coupled to the proximal end of a single-mode optical fiber 624. The distal end of the optical fiber 624 may be threadedly connected or received by a hollow tube 628 of piezoelectric material. The distal end protrudes from the tube 628 as a non-fixed flexible cantilever 632. The piezoelectric tube 628 may be associated with four quadrant electrodes (not shown). The electrodes may, for example, be plated on the exterior, outer surface, or outer perimeter or diameter of the tube 628. A core electrode (not shown) is also located in the core, center, inner perimeter, or inner diameter of the tube 628.

[0065] The driving electronics 636, for example electrically coupled via leads 640, drives the opposing electrode pairs to independently bend the piezoelectric tube 628 in two axes. The protruding distal end of the optical fiber 624 has a mechanical resonant mode. The resonant frequency may depend on the diameter, length, and material properties of the optical fiber 624. By vibrating the piezoelectric tube 628 near the first mechanical resonant mode of the optical fiber cantilever 632, the optical fiber cantilever 632 is caused to vibrate and may sweep through large deflections.

[0066] The tip of the fiber cantilever 632 is biaxially scanned in a filled two-dimensional (2-D) scan area by stimulating resonances in two axes. The light emitted from the fiber cantilever 632 forms an image by modulating the intensity of one or more light sources 620 synchronously with the scanning of the fiber cantilever 632. A description of such an arrangement is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0067] Component 644 of the optical coupler subsystem collimates the light emitted from the scanning fiber cantilever 632. The collimated light is reflected by a reflector 648 into a narrow distributed planar waveguide 612 containing at least one diffractive optical element (DOE) 616. The collimated light is reflected vertically (relative to the wavelength) along the distributed planar waveguide 612 by total internal reflection. Figure 6612) and in doing so repeatedly intersects the DOE 616. The DOE 616 preferably has a low diffraction efficiency. This causes part of the light (e.g., 10%) to be diffracted toward the edge of the larger main planar waveguide 604 at each intersection with the DOE 616, and part of the light to continue its original trajectory down the length of the distributed planar waveguide 612 via TIR.

[0068] At each intersection with the DOE 616, additional light is diffracted toward the entrance of the main waveguide 612. By splitting the incident light into multiple outcoupled groups, the exit pupil of the light is vertically expanded by the DOE 616 in the distribution planar waveguide 612. This vertically expanded light outcoupled from the distribution planar waveguide 612 enters the edge of the main planar waveguide 604.

[0069] Light entering the main waveguide 604 propagates horizontally along the main waveguide 604 via TIR (relative to Figure 6 608). As the light propagates horizontally at least partially along the length of the main waveguide 604 via TIR, the light intersects the DOE 608 at multiple points. The DOE 608 can advantageously be designed or configured to have a phase profile that is a sum of a linear diffraction pattern and a radially symmetric diffraction pattern to produce deflection and focusing of the light. The DOE 608 can advantageously have a low diffraction efficiency (e.g., 10%) so that at each intersection point of the DOE 608, only a portion of the light beam is deflected toward the viewing eye, while the remainder of the light continues to propagate through the waveguide 604 via TIR.

[0070] At each intersection between the propagating light and the DOE 608, a portion of the light is diffracted toward an adjacent facet of the main waveguide 604, allowing the light to escape TIR and exit from the facet of the main waveguide 604. In some embodiments, the radially symmetric diffraction pattern of the DOE 608 additionally imparts a focus level to the diffracted light, both shaping the optical wavefront of a single beam (e.g., imparting curvature) and steering the beam at an angle that matches the designed focal power.

[0071] Thus, these different paths can cause light to be coupled out of the main planar waveguide 604 through multiple DOEs 608 at different angles, focal lengths, and / or produce different fill patterns at the exit pupil. The different fill patterns at the exit pupil can be beneficially used to create a light field display with multiple depth planes. Each layer or group of layers in the stack in the waveguide assembly (e.g., 3 layers) can be used to generate a corresponding color (e.g., red, blue, green). Thus, for example, a first group of three adjacent layers can be used to generate red light, blue light, and green light at a first focal depth. A second group of three adjacent layers can be used to generate red light, blue light, and green light at a second focal depth. Multiple groups can be used to generate a full 3D or 4D color image light field with various depths of focus.

[0072] Example of a biometric exchange operating environment

[0073] Figure 7 7 is a block diagram depicting an illustrative operating environment 700 in which a biometric information exchange system 705 exchanges information between a wearable display system and a computing device capable of acquiring biometric information from a wearer. Biometric information may include information related to the wearer (e.g., Figure 2 For example, physiological characteristics may include, but are not limited to, body posture, eye posture (e.g., eye gaze), pupil dilation state, facial features (e.g., grin or frown), voice characteristics, pulse rate or blood pressure, skin condition (e.g., sweating), etc. For example, behavioral characteristics may include, but are not limited to, gait (physiological characteristics of how the wearer walks), voice characteristics, wearer's tendencies, etc. Physiological and behavioral characteristics may not be mutually exclusive, and in some cases, biometric information may be divided into two categories: physiological and behavioral. For example, the wearer's voice characteristics may not only be physiological characteristics, but may also be affected by the wearer's emotions or feelings. Similarly, the wearer's gait may be affected by the wearer's surroundings, the wearer's companions, the wearer's emotions, etc.

[0074] The biometric information exchange system 705 can be implemented by a computer server, any computing system, or processing module 224, 228. In some embodiments, the server implementing the biometric information exchange system 705 also hosts a website (e.g., an Internet website) that the wearer browses using the wearable display system 710 (e.g., the wearable display system 200). For example, the biometric information exchange system 705 can host the following reference FIG. 8A to FIG. 8EThe display pages 800a-800e and pop-up windows described. Thus, the biometric information exchange system 705 can facilitate the retrieval of biometric information, as well as provide an offer for exchanging biometric information from the biometric information acquirer computing device 715 to the wearer of the wearable display system 710 (e.g., the wearable display system 200). The biometric information exchange system 705 can form a secure connection 725 between the wearable display system 710 and the biometric information acquirer computing device 715 through the network 700. For example, the biometric information exchange system 705 can encrypt any information transmitted over the secure connection 725. Thus, the biometric information exchange system 705 acts as an intermediary server that facilitates secure connections 725 between certain wearable display systems 710 in certain biometric information acquirer computing devices 715. Such a secure connection 725 can maintain confidentiality and facilitate the exchange of biometric information between the wearable display system 710 and the biometric information acquirer computing device 715. In some embodiments, the secure connection 725 allows direct access by the biometric information acquirer computing device 715 and the data storage or processing module (e.g., the local processing and data module 224) of the wearable display system 710, where the wearer's biometric information can be stored. For example, in one embodiment, the biometric information exchange system 705 does not analyze biometrics or perform biometric operations (such as described below with reference to the biometric processing computing system 745), but rather acts as a biometric information channel between the wearable display system 710 and the biometric information acquirer 715 via the secure connection 725. In various embodiments, as further described below, the biometric information exchange system 705 facilitates access to biometric information for the biometric information acquirer computing device 715 via biometric processing results generated using data stored in the wearer data storage 730 or the item catalog 735 (or electronic catalog).

[0075] The biometric information exchange system 705 can be implemented by a separate server that communicates with a server (not shown) hosting a website, or as a web service itself. In this case, the separate server can also act as an intermediary server that facilitates a secure connection 725 between the wearer of the wearable display system 710 and the biometric information acquirer computing device 715. In another embodiment, the information exchange between the wearable display system 710 and the computing device 715 of the potential acquirer of biometric information is distributed between the wearable display system 710, the server hosting the website, and the biometric information exchange system 705. However, for simplicity, the remaining disclosure relates to the biometric information exchange system 705 transmitting a consent request, obtaining biometric information, and generating a biometric information exchange thereto. In such an embodiment, the biometric information exchange system 705 can implement an exchange computing system 740 and a biometric processing computing system 745.

[0076] The exchange computing system 740 and the biometric processing computing system 745 can work in conjunction with each other to facilitate the exchange. For example, the exchange facilitated by the exchange computing system 740 can specify that the biometric information acquirer agrees to process the eye image with the biometric processing computing system 745 to obtain behavioral characteristics about the wearer (e.g., the number of times the wearer glances at a particular product or the type of location where these glances occur). Therefore, by displaying a consent request including the potential exchange content using the exchange computing system 740, the exchange computing system 740 can interact with the wearer of the wearable display system 710. In some embodiments, as further described below, by specifying the type of biometric information that the biometric information acquirer wants to access, the exchange computing system 740 can change the format of the consent request for such a potential exchange to facilitate the exchange. For example, in order to obtain behavioral characteristics, such as the type of location where eye movements occur, the biometric processing computing system 745 can access the GPS unit of the wearable display system 710 or obtain the location of the wearable display system 710, which is determined, for example, by the GPS unit of the wearable display system 710.

[0077] The exchange computing system 740 may implement routines for obtaining various types of biometric information from a wearer using the wearable display system 710, and processing the biometric information to generate a biometric information processing result for the biometric information acquirer computing device 715. For example, as described below, the exchange computing system 740 may implement a routine 900 for exchanging biometric information, which may include displaying a consent request for biometric information using the wearable display system 710, obtaining biometric information from a wearer using the wearable display system 710, and / or processing the biometric information to generate a biometric information processing result for the biometric information acquirer computing device 715.

[0078] When a consent indication has been received from the wearer of the wearable display system 710, the exchange computing system 740 can implement the biometric information exchange based on various factors. As an example, the potential acquirer can use the wearable display system 710 to specify certain categories of biometric information (e.g., physiological or behavioral information), the type of wearer (e.g., a specific demographic), and the exchange price for the biometric information.

[0079] A wearer using the wearable display system 710 may be a wearer associated with a particular demographic. The demographic may specify a type of wearer that frequents a particular retail store that has product items (e.g., a store that offers food product items), a type of wearer that utilizes a particular service (e.g., professional services, hospitality services, etc.), or a type of wearer associated with a particular biographical demographic (e.g., gender, age, place of residence, citizenship, etc.). The biometric information acquirer computing device 715 may specify an exchange of biometric information based on the biometric information or the type of wearer (e.g., the particular demographic of the wearer). In some embodiments, the biometric information acquirer computing device 715 may also specify an exchange price for biometric information acquired from a particular type of wearer.

[0080] The biometric processing computing system 745 can process or obtain various types of biometric information. For example, in various embodiments, the biometric processing computing system 745 implements a routine to obtain an image set from the wearable display system 710, and performs a biometric operation on the associated image set. Such a routine can generate biometric information for use in biometric exchange and / or biometric applications. Various biometric operations can be performed on the image set obtained from the wearable display system 710 to generate various types of biometric information. As an example, an image of the wearer's eye captured by a camera (e.g., an internally facing imaging system 452) and an image of the external world surrounding the wearer captured by a camera (e.g., an externally facing imaging system 454) can be correlated with each other to determine the wearer's eye gaze using the display system. Illustratively, such a correlation can be determined based on a temporal correlation (e.g., based on a timestamp associated with each image). For example, if both the eye image and the external world image have a timestamp of 4:23:42 pm, the two images can be associated with each other based on the temporal correlation. The timestamp may have an accuracy of 1s, 0.1s, 0.01s, 1ms, 0.1ms, 0.01ms, 1ns, or less. Correlations may be determined based on the accuracy of the timestamp. Other correlations are also possible. As another example embodiment, images may be correlated in real time based on a corresponding image acquisition system that simultaneously correlates eye images and external world images. For example, as further described below, dynamic features of an eye such as a dilated pupil may be correlated with an external world image (e.g., captured by an outward-facing imaging system 454) to identify objects of interest to the wearer, which dynamic features of the eye may be obtained from processing the eye image (e.g., captured by one or more eye tracking cameras such as an inward-facing imaging system 452). Additionally or alternatively, the biometric processing computing system 745 may process biometric information or any data obtained from the wearable display system 710 to obtain behavioral features about the wearer, as described below with reference to Figure 8B Further described.

[0081] As another example of performing biometric operations using eye images and external world images, extracting biometric information from eyes in eye images can generally include segmenting a region of each eye image corresponding to the iris of the wearer's eye. Iris segmentation can include the following operations: determining iris boundaries, determining pupil and edge boundaries of the iris, locating the upper or lower eyelid if it occludes the iris, detecting and excluding occlusions by eyelashes, shadows, or reflections, etc. For example, the eye image can be included in an image of the wearer's face, or can be an image of the wearer's eye area. To perform iris segmentation, the boundaries of the pupil (inner boundary of the iris) and the boundaries of the cornea (outer boundary of the iris) can both be determined as separate segments of image data. When used in conjunction with an external world image (e.g., as obtained by an imaging system facing outward from the wearable display system 710), it can be temporally correlated with the eye image to indicate or detect dynamic eye features (e.g., eye movement, eye pattern, eye gaze, eye focus information, pupil dilation, or eye browsing history).

[0082] Further, as another example of processing biometric information (e.g., eye gestures), eye movements of a user of a computer may be tracked. For example, a camera coupled to a monitor of a computer may provide images for identifying eye movements. However, the camera used for eye tracking is some distance from the user's eyes. For example, the camera may be placed on top of a user's monitor coupled to a computer. Therefore, the image of the eye produced by the camera is typically produced with a low resolution. Therefore, processing biometric information from a user may be challenging.

[0083] The eye image may be used for eye gesture recognition. Iris features may be used to identify or authenticate the wearer. As another example, the biometric operation may process the eye image to generate or detect eye movement, eye pattern, eye gaze, eye focus information, pupil dilation, or eye browsing history. As yet another example, the eye image may be used to generate an iris code. The generated iris code may be used to compare with an iris code database storing iris codes of various wearers.

[0084] The biometric information exchange system 705 can be communicatively connected to the wearer data storage 730. The wearer data storage 730 can generally include any repository, database, or information storage system capable of storing biometric information of the wearer. The wearer data can be any type of biometric information, including, but not limited to, eye movements, eye patterns, eye gaze, user activity information (e.g., user past purchase history, eye browsing history, selection through history or click through history, blink through history, rating history, page requests, wish lists and shopping carts added or deleted items, user interface events, marking activities, or combinations thereof, etc.) and user characteristics. User characteristics can include demographic information (e.g., age, gender, geographic location, occupation, income, home address, shipping address, consumption level, interests, hobbies, preferences, settings, or combinations thereof, etc.). The wearer data storage 730 can be at least partially included in the wearable display system 710 or in a data storage associated therewith. For example, the wearable data storage 730 can be included in the local data module 224 and / or the remote data repository 232.

[0085] In other embodiments, the biometric information exchange system 705 utilizes the item data retrieved from the item catalog 735 (as well as, for example, the aforementioned eye movements and eye browsing history) to generate an offer for biometric exchange. The item data for an item may include item details (e.g., name, type, description, price, category, etc.), images, relationships to other items, item ratings, wearer reviews, author pages, user-generated list pages, forum pages, blog pages, etc. The item catalog 735 may also be referred to as an electronic catalog. Such an offer may appear on display pages 800b and 800d generated by the biometric information exchange system 705, as described below with reference to Figure 8B and Fig.8D described.

[0086] The data store of the operating environment 700 may correspond to any physical data store, collection of physical data stores, or virtual data store implemented by one or more physical data stores, such as hard disk drives (HDDs), solid state drives (SSDs), tape drives, network attached storage (NASs), or any other persistent or substantially persistent storage component. Additionally, the item directory 735 and the wearer data store 730 may be located on the same physical data store accessible to the biometric information exchange system 705, or on different physical data stores. In some embodiments, the item directory 735 and the wearer data store 730 may be maintained by the biometric information exchange system 705, or by another website accessible via the network 720, or by one or more web services accessible via various application program interface calls made by the biometric information exchange system 705.

[0087] In addition to the item catalog 735 and the wearer data storage 730, the operating environment 700 may include one or more wearable display systems 710 that communicate with the biometric information exchange system 705 via a network 720. Although the present disclosure is directed to specific examples or descriptions of the wearable display system 710, such as Figure 2 The wearable display system 200 or Figure 4 and Figure 6 The display system 200 shown can utilize various embodiments of the present disclosure through the following: a head-mounted display device capable of obtaining biometric information, and / or capable of displaying an offer to a wearer of the wearable display system 710 (e.g., Figure 8B and Fig.8D The computing devices may include, but are not limited to, portable computers, personal computers, kiosks, thin clients, tablet computers, personal digital assistants (PDAs), hybrid PDA / mobile phones, mobile phones, e-book readers, digital media players, wearable computing devices, integrated components included in computing devices or applications, electronic devices included in vehicles or machinery, gaming devices, set-top boxes, electronic devices included in televisions, and the like. These computing devices may be associated with any of a number of visual, tactile, or auditory output devices, and may be associated with a plurality of input devices for user input, including but not limited to keyboards, microphones, mice, trackballs, trackpads, joysticks, input tablets, tracking points, touch screens, remote controls, game controllers, motion detectors, and the like. In some embodiments, a plurality of devices may be utilized in concert to facilitate displaying quotes to a wearer of the wearable display system 710. For example, a first device (e.g., Figure 2The wearable display system 200 shown may initially display a consent request and provide a response determined by a server hosting a web site and / or implementing the biometric information exchange system 705; and a second device (e.g., a tablet device with a browser-based application) may provide a display of an offer provided by the biometric information exchange system 705. For example, the second device may display FIG. 8A to FIG. 8E Display pages 800a-800e are shown. Thus, references within this disclosure to a user computing device may include multiple computing devices working in concert to facilitate obtaining biometric information and providing an offer to a wearer regarding the biometric information.

[0088] The network 720 may be a wired network, a wireless network, or any combination thereof. The network 720 may be a personal area network, a local area network, a wide area network, a cable network, a satellite network, a cellular telephone network, or a combination thereof. Figure 7 In the illustrative operating environment 700, the network 720 is a global area network (GAN), such as the Internet. Protocols and components for communicating via other aforementioned types of communication networks are known to those skilled in the art of computer communications and thus need not be described in further detail herein.

[0089] Examples of Biometric Exchange Technology

[0090] The biometric exchange technique, routine, or method may be implemented via a wearable display system 710 capable of obtaining biometric information. To obtain biometric information for the biometric exchange routine, a mixed reality display device (MRD) (e.g., Figure 2 The wearable display system 200 or Figure 4 and Figure 6 The eyes of a wearer of the display system 400 shown in FIG. 4 can be imaged using an inwardly facing imaging system 452 (e.g., see FIG. Figure 4 ). The set of eye images obtained by the imaging systems 452, 454 coupled to the wearable display system 710 can be used for biometric information exchange. As described herein, the set of eye images includes eye images from the inward-facing imaging system 452. The outward-facing imaging system 454 can obtain images of the outside world, including images in the direction in which the wearer's eyes are looking. The eye images captured by the inward-facing imaging system 452 can be processed to determine eye pose (e.g., the direction of one or both of the wearer's eyes), generate iris codes, determine pupil size, detect eye patterns, detect eye movement, detect eye gaze, or process eye focus information. Computer vision techniques can be used to process computer vision techniques. The eye images can be stored as part of the eye's browsing history (also referred to as the eye browsing history).

[0091] Illustratively, in the context of a wearable display system 710 (e.g., a wearable head mounted display 200), a camera may be closer to the user's eyes than a camera coupled to a user monitor. For example, a camera may be mounted on a wearable display system 710 that is itself worn on the user's head. The proximity of the eyes to such a camera may result in a higher resolution image of the eyes. Therefore, computer vision techniques may be used to extract features from the user's eyes, particularly features of the iris (e.g., iris features) or features in the sclera surrounding the iris (e.g., sclera features). For example, when captured by a camera close to the eye, the iris of the eye may display a detailed structure. When viewed under infrared lighting, such iris features are particularly evident and may be used for biometric identification. These iris features are considered to be unique from user to user and, therefore, may be used to uniquely identify a user in the manner of a fingerprint. Eye features may include blood vessels in the sclera of the eye (outside the iris), which may also appear particularly evident when viewed under red or infrared light. Such unique iris features viewed at a higher resolution may result in more unique or accurate iris codes being generated for various eye pose images.

[0092] The local processing and data module 224 and / or the remote data repository 232 can store image files, video files, or image audio and video files. For example, in various embodiments, the local processing and data module 224 and / or the remote data repository 232 can store eye images for processing by the local processing and data module 224. The local processing and data module 224 and / or the remote processing module 228 can be programmed to use the biometric information exchange technology disclosed herein in a biometric information exchange, such as to identify or authenticate the identity of the wearer 204 when providing a biometric information exchange, or in obtaining biometric information, such as when a consent indication from the wearer has been received, to process the biometric information for the biometric exchange. Thus, the processing modules 224, 228 can be caused to perform various aspects of the biometric exchange technology. Additionally or alternatively, the controller 450 can be programmed to perform various aspects of the biometric exchange technology.

[0093] Biometric information for biometric information exchange can be processed in various ways. By way of example only, imaging systems 452, 454 can capture video for a particular application (e.g., video of a wearer's eyes for an eye tracking application, or video of a wearer's hands or fingers for a gesture recognition application). The video can be analyzed using biometric information processing techniques by one or both of processing modules 224, 228. For example, processing modules 224, 228 can analyze a set of eye images (e.g., an eye image and a corresponding external world image) to detect eye patterns or eye gazes of a wearer of wearable display system 200. Using this analysis, processing modules 224, 228 can perform biometric information exchange. By way of example, local processing and data module 224 and / or remote processing module 228 can be programmed to store eye images obtained from imaging systems 452, 454. Additionally, the local processing and data module 224 and / or the remote processing module 228 can be programmed to process the image set using the techniques described herein (e.g., routine 900) and transmit biometric information of the wearer of the wearable display system 200 based on the consent category to which the wearer has indicated agreement. In some cases, offloading at least some of the biometric exchange techniques to a remote processing module (e.g., in the "cloud") can increase the efficiency or speed of computation. For example, the remote processing module can perform biometric data operations on the eye image set to obtain biometric information.

[0094] As described above, the biometric information acquirer computing device 715 can obtain biometric information retrieved from the wearable display system 710, which is used by the wearer of the wearable display system 710 in a biometric information exchange. For example, the wearable display system 710 can utilize the biometric information exchange system 705. In order to obtain biometric information in a biometric information exchange performed by the wearer of the wearable display system 710, a consent request can first be transmitted to the wearer of the wearable display system 710. Once the consent request is received, the wearer can respond to the consent request with an indication of the recognition of aspects of the sharing of biometric information permitted by the wearable display system 710 utilized by the wearer. The consent request can be transmitted in a variety of formats or display mechanisms. For example, the wearable display system 710 can display the consent request to the wearer. As another example, the consent request can be transmitted to the wearer of the wearable display system 710 via a display page or content page (e.g., a web page) displayed to the wearer. As FIG. 8A to FIG. 8EAs shown, the consent request may vary in the amount and type of access to the wearer's biometric information that the biometric information acquirer computing device 715 may obtain. As described herein, the biometric information is only shared with the biometric information acquirer computing device 715 utilizing the biometric information exchange system 705 when a response including an indication of consent from the wearer has been received to the transmitted consent request.

[0095] The consent request may include a request for a category of biometric information obtained from the wearable display system 710 for acquisition by the biometric information acquirer computing device 715. For example, a consent request for a set of eye images may be transmitted to the wearer of the wearable display system 710 for acquisition by a specific computing device associated with a specific acquirer. As another example, the wearer may agree to share eye movements obtained from eye images captured by an image capture device coupled to the wearable display system 710, but not agree to share skin sensing detected by the wearable display system 710 (e.g., skin electrical activity detected by a skin electrical sensor). The wearer may have indicated an agreement to the eye movement category, but not to the skin sensing activity category. Additionally or alternatively, the consent request may include a request to access a specific sensor or device coupled to the wearable display system 710 (e.g., via the local processing module 224). For example, the consent request may indicate that access is requested to a GPS sensor to determine location, but specifies that other sensors (e.g., blood pressure sensors) cannot be accessed.

[0096] The consent request may be in various forms or formats. Some consent requests may request that a quote be provided to the wearer of the wearable display system 710 regarding an exchange of biometric information without obtaining any biometric information from the wearable display system 710. Other consent requests may each request that all biometric information about a certain project be shared with a specific biometric information acquirer computing device 715 for a specific exchange price facilitated by the biometric information exchange system 705. By way of example only, the biometric information acquirer computing device 715 may request consent to access acquired biometric information (e.g., biometric information acquired during a certain time period or a specific type of biometric information) via the wearable display system 710. Access to biometric information may be for a specific project. In turn, the biometric information acquirer computing device 715 may provide a promotional item for accessing biometric information about a specific project. Such an example exchange may be referred to as a wearer biometric exchange at a certain exchange price. Additionally or alternatively, the consent requests may each request certain behavioral characteristics of the wearer of the wearable display system 710. Intents regarding actions, such as intent to purchase, can be considered behavioral traits, which can also be exchanged in a biometric exchange, as further described below. Thus, the wearer can be presented with information regarding specific behavioral traits (e.g., intent to purchase a certain type of item) so that the biometric information acquirer can obtain such processed behavioral traits from the wearable display system 710.

[0097] The VR and AR experiences provided by the wearable display system 710 can facilitate the consent request. For example, when the wearer utilizes the wearable display system 710 and views a particular widget, the mixed reality experience can facilitate the consent request. An augmented reality scene including the consent request can be displayed. By way of example only, while a user is viewing a peanut butter jar at a retail store, an augmented reality scene with different brands of peanut butter jars (e.g., a competitor's peanut butter brand) can be displayed in the augmented reality scene. The consent request can indicate that the requested biometric information is an eye image obtained when the user utilizes the wearable display system 710 while viewing the different brands of peanut butter jars. Thus, as described herein, the consent request can take various formats for presentation to the wearer of the wearable display system 710.

[0098] As described below, the wearer can indicate consent to sharing or access of biometric information for the biometric information acquirer computing device 715 by selecting an allow button 808 or icon. For example, in one embodiment, the wearer can blink with an eye pointing to the allow button 808 to select the button, thereby indicating consent to text content associated with the allow button 808. As another example, the wearer can select the allow button 808 using a gesture. One or more computer vision techniques can be used to determine a visual command or gesture such as a blink in a certain direction. Additionally or alternatively, consent requests can be filtered based on certain types of biometric information that the wearer may have indicated interest in sharing or permission to share. For example, only certain pop-up icons regarding corresponding consent categories that the wearer has indicated interest in sharing or permission to share may appear. Other examples of filtering consent requests will be referred to below. Figure 8B Further description.

[0099] As an example, the wearer may apply one or more filters to incoming offers (e.g., "under $5" or "does not require my GPS location," etc.) via an accessible user interface control panel. When an offer arrives, it may be filtered and presented to the wearer only if it satisfies one or more of the filtering conditions. To reduce clutter in the user interface, the system may present the offer to the wearer via a unique icon indicating the presence of an offer. The wearer may select the icon to accept the offer, and the system may automatically obtain (or begin obtaining) biometric information, provide it to the offeror, and compensate the wearer. Such example embodiments are effectively a type of "one-click," "one-permission," "one-select," or "one-interaction" biometric exchange in which the wearer may easily select a filtered offer with minimal interaction. In some such examples, the wearer may not even need to select an icon if the offer satisfies the wearer's filters. As an example, the wearer may pre-filter offers from a supermarket where the wearer shops to allow the supermarket to determine (at appropriate prices) what items the wearer is looking at, which items the wearer actually chooses to purchase (e.g., by placing items in a shopping cart), how much time the wearer spends in different aisles of the store, etc. Whenever the wearer enters the supermarket, the wearable display system 710 can (optionally) display an offer icon and begin the exchange of the wearer's biometric information without further interaction with the wearer. Continuing with this example, an advantage of optionally displaying an offer icon when the wearer enters a location is that the wearer can reject the offer icon by indicating opposition via the wearable display system 710. For example, the wearer can audibly object to the offer (e.g., saying "No, I don't want that") verbally or via a microphone coupled to the wearable display system 710. As another example, the wearer can object to the offer using a visual command (e.g., blinking toward the "Don't Allow" button 812). Thus, the wearer can disallow access to the biometric information for that particular visit to the location, while the exchange of the wearer's biometric information in subsequent visits to the location can still begin without further interaction with the wearer if the wearer does not object to the displayed offer icon during one or more subsequent visits.

[0100] The computer vision module may implement one or more computer vision techniques. The computer vision module may determine visual commands of the wearer, such as blinking in a certain direction or any direction, or gestures (e.g., with one or both hands, or one or more objects). The computer vision module may determine the object, person, or scene that the wearer is looking at. Non-limiting examples of computer vision techniques include: Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Rapid Retinal Keypoints (RREAK), Viola-Jones algorithm, Eigenface method, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) technology, Sequential Bayesian estimator (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive threshold (and other threshold techniques), Iterative Closest Point (ICP), Semi-Global Matching (SGM), Semi-Global Block Matching (SGBM), feature point histogram, various machine learning algorithms (such as support vector machine, k-nearest neighbor algorithm, Naive Bayes, neural network (including convolutional or deep neural network) or other supervised / unsupervised models, etc.), etc.

[0101] The speech recognition module may implement one or more speech recognition techniques. The speech recognition module may determine the wearer's verbal or audio command (e.g., saying "yes" or "no"). The speech recognition technique may be based on machine learning, such as long short-term memory (LSTM) recurrent neural networks (RNNs). The speech recognition technique may be based on hidden Markov models (HMMs), dynamic time warping (DTW)-based speech recognition, machine learning, or end-to-end automatic speech recognition. The machine learning technique for speech recognition may be based on neural networks, such as long short-term memory (LSTM) recurrent neural networks (RNNs), time delay neural networks (TDNNs), or deep forwarding neural networks (DNNs) or recurrent neural networks (RNNs).

[0102] Speech recognition techniques and computer vision techniques may be additionally or alternatively performed by various machine learning algorithms. Once trained, the machine learning algorithm may be stored by the wearable display system 710. Some examples of machine learning algorithms may include supervised or unsupervised machine learning algorithms, including regression algorithms (such as ordinary least squares regression), instance-based algorithms (such as learning vector quantization), decision tree algorithms (such as classification and regression trees), Bayesian algorithms (such as naive Bayes), clustering algorithms (such as k-means clustering), association rule learning algorithms (such as prior algorithms), artificial neural network algorithms (such as perceptrons), deep learning algorithms (such as deep Boltzmann machines or deep neural networks), dimensionality reduction algorithms (such as principal component analysis), integration algorithms (such as cascade generalization) and / or other machine learning algorithms. In some embodiments, a single model can be customized for a single data set. For example, a wearable device can generate or store a basic model. The base model can be used as a starting point to generate additional models specific to a data type (e.g., a specific user in a telepresence session), a data set (e.g., an additional set of images obtained by a user in a telepresence session), a conditional situation, or other variations. In some embodiments, the wearable display system 710 can be configured to utilize a variety of techniques to generate models for analysis of aggregated data. Other techniques can include the use of predetermined thresholds or data values.

[0103] Example display page

[0104] For ease of illustration in the figure, the pseudonym "biometricexchange.com" is used in the displayed page to refer to the website hosted by the biometric information exchange system 705. In addition, the pseudonyms William Wearer and Adam Acquirer are used to refer to the specific wearer of the wearable display system 710 and the biometric information acquirer computing device 715, respectively.

[0105] FIG. 8A to FIG. 8E An example of the format and content of a representative display content page or display page (e.g., a web page) is shown, in which the biometric information exchange system 705 exchanges information between the wearable display system 710 and a computing device of a potential acquirer of the wearer's biometric information. For purposes of illustration, the representative display pages are presented as they might appear in, for example, a web browser displayed by the wearable display system 710. Although FIG. 8A to FIG. 8EThe example shown is represented as a web page, but this is for illustration and is not intended to be limiting. In other embodiments, the content of the biometric information exchange system 705 can be presented by the wearable display system 710 in other formats, such as via an application (e.g., an "app") that can be executed on the processing modules 224, 228. For example, in one embodiment, a pop-up window (or an alert, a modal / dialog, a flash notification / sound notification, a light box / theater, a pop-up notification / prompt box / hover card, or any combination thereof) can be presented to the wearer, which pop-up window includes a portion of the content of the biometric information exchange system 705 (e.g., an icon indicating a request for consent for biometric information). In some embodiments, the content of the display page can be seamlessly integrated into the scene shown to the wearer (e.g., to the wearer). Figure 2 The wearable display system 200 or Figure 4 and Figure 6 An augmented reality scene 100 is shown to a wearer of the display system 400 ).

[0106] Fig. 8A An example display page 800a generated by the biometric information exchange system 705 for a wearer of the wearable display system 710 to provide consent for an offer to view biometric information is shown. As shown, even before accessing the display page 800a of biometricexchange.com, the wearer may provide consent to the biometric information exchange system 705 to connect to the system. Fig. 8A 800a. In this illustrative example, William may be presented with a pop-up window 804 that may be displayed on top of the display page 800a, requesting permission from William to connect his wearable display system 710 to the biometric information exchange system 705. The pop-up 804 describes the actions that the biometric information exchange system 705 may take if William agrees to connect his wearable display system 710 to the biometric information exchange system 705. If William decides not to agree, William may click or select a do not allow button 812, and the biometric information exchange system 705 may not attempt to connect or link William's wearable display system 710 to the biometric information exchange system 705. In this case, the wearer's social network information remains private and unavailable to the biometric information exchange system 705.

[0107] If William decides to agree, William can select the allow button 808 to indicate the consent of the aspect of the biometric information. As shown, in this example, William indicates that he agrees to receive quotes for his biometric information and biometric information exchange. By selecting the allow button 808, William provides an indication of consent to the quote for the biometric information exchange and can access biometricexchanage.com. If William does not have or has not yet created an account with the biometric information exchange system 705, the biometric information exchange system 705 can prompt William to create an account on the biometric information exchange system 705. For example, the biometric information exchange system 705 can use a pop-up window (not shown) that allows William to enter his authentication credentials (e.g., username and password or iris code) so that the biometric information exchange system 705 can be authorized to access William's biometric information from the wearable display system 710. The pop-up box 804 can also include a description of the policy of the biometric information exchange system to protect the privacy of the wearer's personal data, as well as a description of how the wearer can use the functions of the allow button 808 and the disallow button 812.

[0108] In some implementations, once a consent indication is determined from a response by the wearer of the wearable display system 710, the VR, AR, or MR experience provided by the wearable display system 710 can facilitate the presentation of an offer for exchange of biometric information. For example, when William utilizes the wearable display system 710 and views a particular widget, the AR experience can facilitate a consent request. An augmented reality scene including a consent request can be displayed. But as an example only, when a user is viewing items at a physical location, an augmented reality scene with similar items but of different types (e.g., items of different brands) can be displayed in the augmented reality scene. When William may be utilizing the wearable display system 710 for viewing these types of items, the consent request can indicate a request for biometric information of the eye image obtained. In another embodiment, as described below with reference to Figure 8C As further described, the consent request may indicate that only one aspect of the biometric information is to be processed based on the acquired eye image. For example, the consent request may indicate that eye patterns when viewing those specific types of items may be transmitted to the biometric information exchange system 705, receiving an offer for that biometric information in exchange for a certain price.

[0109] Figure 8B It is shown that when the wearer agrees to watch Fig. 8AWhen William starts to receive a quote provided by the biometric information exchange system 705, an example display page 800b generated by the biometric information exchange system 705 and a corresponding pop-up window that can be displayed on the example display page 800b. After the connection is established by the biometric information exchange system 705, a pop-up window 8216 indicating that the connection has been successfully established can be presented to William. In one embodiment, the biometric information exchange system 705 can use an API call to the processing modules 224, 228 of the wearable display system 710 to establish a connection between William's wearable display system 710 and the biometric information exchange system 705. For example, with William's consent, the biometric information exchange system 705 can use William's authentication credentials (e.g., username and password) to automatically log in to the biometric information exchange system 705 for William. In another embodiment, if William has a profile associated with the wearable display system 710 stored in the data repository 232, the biometric information exchange system 705 can retrieve William's profile from the data repository 232 and utilize the profile for providing the biometric exchange offer. Figure 7 As described, the biometric information exchange system 705, which may be hosted by a server computing device, may form a secure connection 725 between the wearable display system 710 and the biometric information acquirer computing device 715. Thus, the secure connection 725 hosted by the biometric information exchange system 705 may facilitate the exchange of biometric information between the wearer of the wearable display system 710 and the biometric information acquirer computing device 715.

[0110] Using a secure connection 725 to the biometric information exchange system 705, the display page 800b can display a quote 820 for biometric information exchange. As shown, the quote 820 includes a description of the biometric information obtained by the wearable display system 710 for certain biometric exchange prices. As an example quote 820, the eye browsing history of a specific item at the example big box store (or any retail institution or showroom, whether online or physical) can be exchanged with the biometric information acquirer computing device 715 at an exchange price of $10 per visit. Therefore, when the eye browsing history related to the example item is obtained by the wearable display system 710, each visit of William to the big store using the wearable display system 710 can be exchanged for $10 on the biometric information exchange. As another example quote 820, when William uses the wearable display system 710 while driving on US-101, William's eye gaze while driving on US-101 can be exchanged for an exchange price of $99 per month. Thus, William can view the offer 826 displayed on display page 800b without agreeing to a specific exchange. FIG. 8C to FIG. 8EAs shown, William can agree to different types of consent based on the consent request for biometric information from the biometric information exchange system 705. Figures 8A to 8B As shown, William has agreed to a basic level of consent that only allows the biometric information exchange system 705 to provide William with a quote regarding William's biometric information.

[0111] Displaying the quotes 820 on the display page 800b may be implemented in conjunction with filters or different display settings. For example, William may select a filter ( Figure 8B ) to automatically exclude certain offers. For example, William may select a filter that automatically excludes offers below $1. As another example, William may select a filter that automatically excludes offers related to his eye gaze. In one embodiment, William may be able to see offers that he is not actually eligible to select. For example, when driving on US-101, he may be able to see offers for eye gaze, even if William does not live in an area near US-101. In this case, such offers may have been provided by the biometric information acquirer computing device 715, which seeks to acquire biometric information related to the wearer driving on US-101. Additionally or alternatively, filters may be used to include certain offers that the user may be interested in selecting.

[0112] As another display setting, Figure 8B Instead of the displayed U.S. dollar price, the exchange price can be displayed in a different currency. For example, the exchange price can be expressed in some other transaction medium (e.g., reward points) specified by the biometric information exchange system 705. In this case, the other currency can be used for various other exchanges on the biometric information exchange system 705 without actually obtaining the currency in U.S. dollars through a financial medium. Therefore, the biometric information exchange system 705 can perform exchanges in an exchange medium customized for biometric information exchanges.

[0113] Various biometric information exchange offers are possible. As William utilizes the wearable display system 710, the biometric information acquirer can provide various offers to William the wearer via the biometric information exchange system 705. As described above, the wearable display system 710 can obtain eye images to detect or process dynamic eye features. Dynamic eye features can include eye movements, eye patterns, eye gaze, eye focus information, pupil dilation state, or eye browsing history. In addition, external world images (e.g., captured by the imaging system 454) can also be used for various offers based on demographics or locations. For example, when an image from the wearable display system 710 obtained in a retail store can indicate the demographics of the retail store. As just one example, an image of a retail store when the wearer is browsing can indicate that most of the people in the retail store are under 30 years old. Therefore, the biometric information acquirer computing device 715 that has obtained such an image can supply items that are popular with the age demographics in the retail store. As another example, the demographics of people browsing a particular department in a retail store can also indicate the interest level of the demographics for the type of item itself. For example, images obtained from a wearer in a retail store during the afternoon hours may indicate that men primarily browse the grocery section during that time period. Thus, the biometric information acquirer computing device 715, upon obtaining these images, may provide advertisements or offers 820 to men who were in the grocery section of that retail store during that time period. The biometric information acquirer computing device 715 may use the biometric information for related biometric information activities or projects. Thus, the systems described herein may provide eye images and / or external world images to the biometric information acquirer computing device 715, which utilizes the biometric information exchange system 705 to connect with the wearer's associated wearable display system 710 (e.g., wearable display system 200).

[0114] The wearable display system 200 using the processing modules 224, 228 or the biometric information exchange system 705 can process biometric information obtained from a wearer using the wearable display system 200. For example, an eye image can be processed to obtain biometrics about the wearer. As an example of processing an eye image, the eye image can be analyzed for action intention based on pupil dilation, eye pointing, or an item that the eye is looking at. Illustratively, when a user is viewing multiple peanut butter jars in a retail store (e.g., as analyzed from an image group associated with the item), the duration of time that the eye is looking at a certain brand of peanut butter jar can indicate a purchase intention. The wearer's pupil dilation can confirm the intention to purchase the peanut butter jar, for example, indicating that the wearer's pupil is dilated when viewing the peanut butter jar. Or as another example, after viewing the initial peanut butter jar for a period of time, the eye image can indicate that the eye has moved to another different brand of peanut butter jar. For example, the corresponding external world image can be used to associate the eye gaze with a specific item viewed by the wearer. This pattern of eye gaze can indicate that the initial peanut butter jar can be purchased. Purchase data from a registry point-of-sale transaction can indicate that the purchase actually occurred. Conversely, final purchase data can be used to infer the wearer's intent in purchasing the peanut butter jar. For example, the final purchase of an initial peanut butter jar can indicate that the wearer has a higher probability of viewing the final purchased item for a longer duration than a competitor brand of the same item brand. Biometric information obtained about the wearer's skin (e.g., detected by a galvanic skin sensor system coupled to the wearable display system 710) can also be used to infer intent for actions (e.g., behavioral characteristics) for the item. As an example of such a galvanic skin sensor system, detection of the wearer's sweat by the galvanic skin sensor system can indicate that the wearer is interested in purchasing the item at a retail store.

[0115] Although this example has been described with reference to analyzing images of peanut butter jars in a retail store and detecting sweat from a wearer, additional biometrics of the wearer may be inferred or processed from data acquired by the wearable display system 710. In various embodiments, any data acquired from the wearable display system 710 may be captured, inferred, processed, or behavioral characteristics may be determined. For example, as described above with reference to Figure 2 As described, the data acquired by the local processing module 224 may include data captured from sensors (which may, for example, be operably coupled to the frame 212 or otherwise attached to the wearer 204), including sensors such as image capture devices (such as, a camera), microphones, inertial measurement units, accelerometers, compasses, gyroscopes, position sensors (e.g., GPS sensors), radios, voice detectors, galvanic skin sensors, pulse or blood pressure sensors, electrocardiogram sensors and / or electroencephalogram sensors.

[0116] The correlation of information obtained from the processing of the eye image, the external world image, and any other data obtained from the wearable display system 710 can be used to determine or infer the behavioral characteristics of the wearer (e.g., purchase items). As another example of processing behavioral characteristics from eye images, eye images indicating that the wearer is entering a car and eye images indicating that the wearer has fastened their seat belts can both be processed to determine the time it takes the wearer to put on the seat belt after entering the car. This time for seat belt action can be exchanged with automobile manufacturers, automobile safety organizations, automobile insurance companies, or motor vehicle departments as behavioral characteristics of the wearer. Additionally or alternatively, in an aggregated manner, various wearer behavioral characteristics can be processed or inferred from the aggregated data obtained from several wearable display systems 710. Such aggregated behavioral characteristics can also be processed or analyzed to obtain behavioral characteristics for certain demographics or certain types of wearers (e.g., family vehicle drivers or sports vehicle drivers). In some cases, the entity requesting biometric information can provide a processing algorithm or software (which can remain proprietary to the entity) to the biometric information exchange system 705, which uses the processing algorithm or software on behalf of the entity to process the aggregated biometric information.

[0117] In some embodiments, the signage associated with the item may be the focus of the analysis, rather than the item itself. For example, a large retailer may place two logo signs with different formats and test (e.g., using A / B testing) in two different stores which sign receives more interest from customers. Continuing with this example, according to the methods described herein, since various wearers have agreed to provide biometric information via consent requests, the retailer can obtain aggregate dynamic eye features from the wearers to determine that one of the signs receives more interest from customers than the other. Thus, the retailer can determine (e.g., using A / B testing or other statistical techniques) that a particular sign with a particular format is more popular than another type of sign with a different format. Additionally or alternatively, the signage associated with the item and the item itself can be used to indicate an action intent for the item. In both cases, the set of images associated with the item can be processed by the biometric information exchange system 705 or the processing modules 224, 228 to provide an offer 820 to the wearer of the wearable display system 710 for biometric information exchange, as displayed on the display page 800b. For example, as described below with reference to Fig. 9 As described, processing modules 224, 228 may implement routine 900 to process biometric information from an image set including an eye image and a corresponding external world image.

[0118] Although the foregoing example has been described in the context of analyzing external-world images of peanut butter jars at a retail store, this is for illustration only and is not intended to be limiting. In other embodiments, any item may be used in the analysis of the obtained eye images. Additionally, any correlation of information obtained from the processing of the eye images and the external-world images may be used to infer action intent for the item (e.g., to purchase the item). For example, by correlating eye focus information and eye movements of wearers in the same retail store with each wearer's corresponding external-world images, offers 820 may be made to various wearers as the information becomes more generally available. As another example, the exchange price for a particular type of biometric information may be reduced when the correlation indicates that such information is not useful, or increased when the correlation indicates that such biometric information is useful. Other factors may also affect the offer 820. For example, if a large number of wearers select a particular offer regarding eye gaze while driving on US-101, the exchange price may be reduced to $5 per month instead of $5. Figure 8B The monthly price is $99. Fig. 9 As further described in , routine 900 depicts an example workflow for exchanging biometric information, such as displaying a consent request associated with biometric information using the wearable display system 710, obtaining biometric information from a wearer using the wearable display system 710, and / or processing biometric information to generate a biometric information processing result for a biometric information acquirer computing device 715.

[0119] After having opted in and authorized the connection between the wearable display system 710 and the biometric information exchange system 705, the wearer may change his or her mind and decide to opt out of the connection to the biometric information exchange system 705. For example, the biometric information exchange system 705 may generate a display page (or a link thereto) so that William may withdraw consent and disassociate William's wearable display system 710. Upon withdrawing the wearer's consent, the biometric information exchange system 705 may remove the wearer's authentication information (e.g., username and password information), profile information, and any biometric information obtained from any data repository or cache associated with the biometric information exchange system 705.

[0120] Figure 8C An example display page 800c generated by the biometric information exchange system 705 for the wearer is shown, which provides a consent request for receiving a quote regarding certain categories of the wearer's biometric information. As illustrated, even before the biometric information is shared with biometricexchange.com, during the process of the wearer providing consent for the biometric information exchange system 705 to connect to the system, Figure 8CA pop-up window is displayed on the display page 800c of the display page 800c. In this illustrative example, William may be presented with a pop-up window 824 that may be displayed on the display page 800c, which pop-up window 824 requests William's permission to connect his wearable display system 710 with the biometric information exchange system 705 and share certain biometric information. Pop-up 824 describes the actions that the biometric information exchange system 705 may take if William agrees to connect his wearable display system 710 with the biometric information exchange system 705 and share certain aspects of his biometric information with the biometric information exchange system 705. For example, William may select certain categories (e.g., eye movements and eye patterns) that may be shared with third-party biometric information acquirers connected to the biometric information exchange system 705. If William decides not to agree, William may click or select the do not allow button 812, and the biometric information exchange system 705 may not attempt to connect or link William's wearable display system 710 to the biometric information exchange system 705. In this case, the wearer's biometric information remains confidential and unavailable to the biometric information exchange system 705.

[0121] If William decides to agree, William can select the allow button 808 to indicate his agreement with respect to the biometric information exchange. As shown, in this example, William indicates that he agrees to share certain categories of his biometric information for the supply of the biometric information exchange. By selecting the allow button 808, William has provided an indication of consent related to certain categories of offers for the exchange of biometric information, and the wearable display system 710 can share such biometric information with biometricexchanage.com. As described above, the pop-up window 824 can also include a description of the biometric information exchange system's policy for protecting the privacy of the wearer's personal data, as well as a description of how the wearer can use the allow button 808 and the do not allow button 812 functions.

[0122] Fig.8D An example display page 800d generated by the biometric information exchange system 705 is shown, as well as Figure 8C Initiated by, a corresponding pop-up window may be displayed on the example display page 800d when the wearer agrees to receive quotes regarding certain categories of biometric information using the biometric information exchange system 705. After the connection is established by the biometric information exchange system 705, a pop-up window 828 may be presented to William indicating that the connection has been successfully established. As described above with reference to Figure 8B As described, a connection to the biometric information exchange system 705 may be established upon receipt of consent from the pop-up window 828 .

[0123] In the event of a successful connection with the biometric information exchange system 705, the display page 800d may display an offer 832 for the biometric information exchange. As shown, the offer 832 includes a description of the biometric information obtained by the wearable display system 710 at a certain biometric exchange price. As an example offer 832, for every 10 visits to the big box store, eye movements associated with a toothpaste brand at an example big box store (or any retail establishment or showroom, whether online or physical) may be exchanged with the biometric information acquirer computing device 715 for an exchange price for a promotional item similar to the toothpaste brand (e.g., a competitor or promotional toothpaste product). For example, during William's visit to the big box store, Adam acquirer may want to exchange eye movements of a competitor wearable device for a promotional wearable watch. As another example offer 832, when utilizing the wearable display system 710 while William is driving on US-101, after 90 minutes of driving on US-101, William's eye pattern may be exchanged for an exchange price of a free 15-minute ride offered by an example taxi company. Thus, if FIG. 8C to FIG. 8D As shown, William has agreed to share certain categories of biometric information based on an initial consent request from the biometric information exchange system to share William's biometric information. In turn, the biometric information exchange system 705 facilitates offers for William's biometric information at an exchange price, as supplied by various biometric information acquirer computing devices 715. Fig.8D As another example, not depicted, William may be presented with a list of offers associated with behavioral characteristics of the wearer of the wearable display system 710 at corresponding exchange prices for exchange over the biometric information exchange system 705 .

[0124] Fig. 8EAn example display page 800e is shown, which can be generated by the biometric information exchange system 705 to provide a consent request for sharing some or all of the wearer's biometric information. In this illustrative example, William can be presented with a pop-up window 836 that can be displayed above the display page 800e, which pop-up window 836 requests William's permission to connect his wearable display system 710 to the biometric information exchange system 705 and share all of his biometric information. The pop-up window 836 describes the actions that the biometric information exchange system 705 can take if William agrees to connect his wearable display system 710 to the biometric information exchange system 705 and share any of his biometric information with the biometric information exchange system 705. For example, the wearable display system 710 utilized by William can share biometric information (e.g., eye movements and eye patterns) with third-party biometric information that needs to be connected to the biometric information exchange system 705. This biometric information can be used by the biometric information exchange system 705 to provide William with an offer for any aspect of his biometric information by the Adam acquirer. Therefore, if William agrees with the text content associated with the pop-up window 836, William can select the allow button 808. As described above, the pop-up window 836 can also include a description of the policy of the biometric information exchange system to protect the privacy of the wearer's personal data, and a description of how the wearer can use the allow button 808 and the do not allow button 812 functions.

[0125] However, if William decides not to consent, William may click or select the do not allow button 812, and the biometric information exchange system 705 may not attempt to connect or link William's wearable display system 710 to the biometric information exchange system 705. In this case, the wearer's biometric information remains private and unavailable to the biometric information exchange system 705.

[0126] Example Biometric Information Exchange Routine

[0127] Fig. 9 A flow chart of an illustrative biometric information exchange routine is shown. Routine 900 depicts an example workflow for displaying a consent request for biometric information with wearable display system 710, obtaining biometric information from a wearer using wearable display system 710, and processing the biometric information to generate a biometric information processing result for biometric information acquirer computing device 715. Routine 900 can be performed by biometric information exchange system 705.

[0128] At block 904, a request for biometric information is received from the biometric information acquirer computing device 715. The request for biometric information may be received from a variety of sources, including, but not limited to: a server hosting the biometric information exchange system 705, a biometric information acquirer computing device 715 directly connected to the wearable display system 710, a computing device hosting the biometric information exchange system 705, or any computing device associated with the biometric information acquirer computing device 715 and connected to a network to which the wearable display system 710 is also connected.

[0129] Continuing with routine 900, at block 908, a consent request is displayed with the wearable display system 710. As described above, the wearable display system 710 is a type of wearable display system 710. In various implementations, the consent request may be displayed on a display of the wearable display system 710, as described above with respect to examples of biometric information exchange techniques. For example, the consent request may be displayed in an augmented reality scene on the wearable display system 710 that displays competitor items if the items are included in an offer for biometric information exchange.

[0130] At decision block 912, a determination is made as to whether a consent indication has been received. The consent indication may be regarding aspects of biometric information. For example, a consent response may be received in response to display of a consent request. The consent response may include a consent indication that is associated with an assent to aspects of biometric information. Illustratively, the consent response may be regarding several categories of biometric information, with the wearer of the wearable display system 710 assent to only some of those categories. In this case, the consent indication is for those assent categories that were assent to. On the other hand, the consent indication may be full consent for any or all biometric information received, captured, or detected by the wearable display system 710 when the wearer to whom the consent request has been displayed utilizes the wearable display system 710.

[0131] In some cases, the consent indication may not be received. For example, after a certain period of time when the consent request is displayed, the consent indication may not be received. Alternatively or additionally, the consent response may indicate that no consent is given to any biometric information. For example, the wearer may not indicate consent to any consent category of biometric information. If the user does not indicate consent to the consent request, the routine 900 proceeds back to box 904. Additional consent requests may be repeatedly received and displayed to the wearer. As just one example, the consent request may not be displayed to the wearer again for a period of time. As another example, at box 908, a different consent request with a narrower scope may be displayed to the wearer using the wearable display system 710. For example, the narrower consent request may be a request to display only an offer to share biometric information (in which case, as described below, box 920 may not be part of routine 900). In any case, the routine 900 may repeatedly proceed to decision box 912 until a consent indication related to an aspect of the biometric information has been affirmatively responded to. In some implementations, a consent indication can be determined based on a lack of rejection of a number of consent requests (eg, 2, 3, 4, 5, 10, or more) shown to the user.

[0132] If at decision block 912, it is determined that an indication of consent has been received in a consent response from the wearer, the routine 900 proceeds to block 916. For example, in various embodiments, when the user is provided with a consent indication as described above with reference to FIG. Fig. 8A , Figure 8C and Fig. 8E When the display pages 800a, 800c, 800e described in the illustrated example receive an "allow" indication, it can be determined that a consent indication has been received regarding the biometric information.

[0133] Continuing with routine 900, at block 916, biometric information is obtained using the wearable display system 710. As described above, the biometric information may be obtained using various devices such as an image capture device or a galvanic skin sensor system coupled to the wearable display system 710. The biometric information may include an eye image or the result of processing an eye image, such as eye focus information.

[0134] At block 920, based on the initial request for biometric information and the wearer's indication of consent, the obtained biometric information (or all of it) is transmitted to the biometric information exchange acquirer 715. For example, in various embodiments, as described above with reference to Fig.8DAs described in the illustrated example, when the wearer has connected to the biometric information exchange system 705 to receive quotes for certain aspects of biometric information at various exchange prices proposed by the biometric information exchange acquirer 715, portions of the biometric information obtained regarding the biometric information category can be transmitted to the biometric information exchange acquirer 715. In one embodiment, the biometric information transmitted is dependent on or based in part on an initial request from the biometric information acquirer computing device 715 (e.g., a request for consent displayed to the wearer) and an indication of consent that the wearer has indicated (e.g., only quotes related to eye patterns for items are displayed to the wearer).

[0135] At block 924, compensation may be provided to the wearer for the exchange of biometric information. The compensation may include monetary compensation, discounts or rebates on goods or services, preferences for goods or services (e.g., preferred tee times at a golf course), or any other type of compensation acceptable to the provider of the biometric information. The compensation may be provided to the provider of the biometric information (e.g., the wearer of the wearable display system 710) or a designee of the provider. The compensation may be provided by the biometric information exchanger, the biometric information acquirer, or a third party.

[0136] At block 924, the biometric information exchange may receive payment for the biometric information exchange services provided. For example, the biometric information exchange may charge the biometric information acquirer a fee for connecting the acquirer to the biometric information provider. Thereafter, the routine 900 ends.

[0137] In some implementations, alternatively or additionally, a quote regarding the biometric information can be transmitted to the wearable display system 710 based on the transmitted biometric information. Figure 8B and Fig.8D As shown, offers 826 and 838 may be communicated to the wearer by displaying such offers on display pages 800b, 800d. The routine 900 may provide such offers to the wearer to facilitate exchange via the biometric information exchange system 705.

[0138] In various embodiments, routine 900 may be performed by a hardware processor (e.g., processing module 224, 228 or controller 450) of a display system such as in embodiments of display system 200. In other embodiments, a remote computing device with computer-executable instructions may cause wearable display system 710 to perform aspects of routine 900. For example, the remote computing device may determine that the consent indication is regarding an aspect of biometric information (e.g., an eye pattern).

[0139] In some embodiments, a method is performed under the control of a hardware processor, and the method includes: transmitting a consent request to a user, the request including multiple consent categories for the user's biometric information, the biometric information including physiological information, behavioral information, or any combination thereof; receiving a response to the consent request; determining that the response includes a consent indication from the user, an consent indication regarding a consent category that the user has indicated agreement with; obtaining multiple eye images of the user; and in response to a request for biometric information from a biometric information acquirer computing device, sending the multiple eye images to the biometric information acquirer computing device based at least in part on the consent categories that the user has indicated agreement with and the request for biometric information.

[0140] In some embodiments, a method is performed under the control of a hardware processor, and the method includes: causing a consent request regarding a user's biometric information to be displayed; determining that a response to the consent request includes a consent indication regarding an aspect of the biometric information; obtaining a biometric information processing result based on the biometric information processing; and providing the biometric information processing result to a biometric information acquirer computing device associated with the consent indication.

[0141] In some embodiments, a method is executed under the control of a hardware processor, and the method includes: receiving an indication by a wearer of a head-mounted display that consents to exchange with a biometric information acquirer a category of biometric information that can be obtained from the wearer of the head-mounted display; obtaining biometric information of the wearer of the head-mounted display; and transmitting biometric information exchange data associated with the obtained biometric information to the biometric information acquirer.

[0142] Additional aspects

[0143] In a first aspect, a head-mounted display system is disclosed. The head-mounted display includes: a first image capture device configured to capture multiple eye images of an eye; a second image capture device configured to capture multiple external world images; a display configured to display virtual images to the eyes of a wearer of the head-mounted display; a hardware processor programmed to: transmit a consent request to the wearer of the head-mounted display, the request including multiple consent categories for the wearer's biometric information; receive a response to the consent request; determine that the response includes a consent indication from the wearer of the head-mounted display, the consent indication being associated with a consent category that the wearer has indicated as agreeing to; obtain multiple eye images of the wearer of the head-mounted display captured by the first image capture device; and in response to a request for biometric information from a biometric information acquirer computing device, transmit the multiple eye images to the biometric information acquirer computing device based at least in part on the consent categories that the wearer has indicated as agreeing to and the request for biometric information.

[0144] In a second aspect, according to the head-mounted display system of aspect 1, the multiple consent categories for the wearer's biometric information include at least one of a quotation category, a partial sharing category, a full sharing category, or any combination thereof.

[0145] In a third aspect, according to the head mounted display system of aspect 2, the consent category to which the wearer has indicated assent corresponds to a full sharing category.

[0146] In a 4th aspect, according to any one of aspects 2 to 3, the head mounted display system, wherein the partial sharing category includes sharing of multiple eye images about item brands or item products in a retail store.

[0147] In a 5th aspect, according to any one of aspects 1 to 4, the head-mounted display system, wherein the hardware processor is further programmed to: perform biometric data operations on multiple eye images to obtain biometric information, wherein, in order to transmit the multiple eye images in response to a request for biometric information from a biometric information acquirer computing device, the hardware processor is further programmed to: transmit a portion of the biometric information obtained based at least in part on a consent category that the wearer has indicated as agreeing to in response to the request for biometric information from the biometric information acquirer computing device.

[0148] In a 6th aspect, according to the head-mounted display system of aspect 5, the biometric information corresponds to at least one of eye patterns, eye movements, eye gazes, or eye browsing history.

[0149] In aspect 7, a head mounted display system as described in any one of aspects 5 to 6, wherein, in order to perform biometric data operations on multiple eye images to obtain biometric information, the hardware processor is further programmed to: perform pose estimation or iris code generation using multiple eye images.

[0150] In an eighth aspect, the head mounted display system of any one of aspects 1 to 7, wherein the hardware processor is further programmed to: display the consent request using the head mounted display system.

[0151] In aspect 9, a head mounted display system according to any one of aspects 1 to 8, wherein the hardware processor is further programmed to: in response to receiving a response to the consent request, display multiple biometric exchange offers using the head mounted display, wherein the multiple biometric exchange offers include offers associated with biometric information of items from an electronic catalog or offers associated with biometric information based on the location of a wearer of the head mounted display system.

[0152] In a 10th aspect, the head mounted display system of aspect 9, wherein the hardware processor is further programmed to: filter the plurality of biometric exchange offers based on a threshold exchange price.

[0153] In an 11th aspect, according to any one of aspects 9 to 10, the head mounted display system, wherein the hardware processor is further programmed to: filter the plurality of biometric information exchange offers based on exclusions of aspects of the biometric information, wherein the exclusions are specified by a wearer of the head mounted display system.

[0154] In aspect 12, the head mounted display system of any of aspects 1 to 11, wherein the hardware processor is further programmed to filter the plurality of biometric exchange offers based on inclusion of aspects of the offers, wherein the inclusion is specified by a wearer of the head mounted display system.

[0155] In a 13th aspect, according to any one of aspects 1 to 12, the head mounted display system, wherein the hardware processor is further programmed to: form a secure connection between the head mounted display system and the biometric information acquirer computing device.

[0156] In aspect 14, a head-mounted display system according to any one of aspects 1 to 13, wherein the hardware processor is further programmed to: process multiple eye images captured by the first image capture device and multiple external world images captured by the second image capture device to obtain behavioral characteristics about the wearer of the head-mounted display system.

[0157] In a 15th aspect, according to the head-mounted display system of aspect 14, wherein the behavioral characteristics about the wearer include, in a plurality of eye images determined using a plurality of external world images, item brands or item products in a retail store viewed by the wearer's eyes.

[0158] In a 16th aspect, according to any one of aspects 14 to 15, the hardware processor is further programmed to: display a second consent request regarding the obtained behavioral characteristics using the head mounted display system.

[0159] In aspect 17, the head-mounted display system of aspect 16, wherein the hardware processor is further programmed to: in response to receiving a response to a second consent request regarding the obtained behavioral characteristics, display a second plurality of biometric exchange offers using the head-mounted display, wherein the second plurality of biometric exchange offers include: offers associated with biometric information about items from the electronic catalog, offers associated with biometric information based on the location of the wearer of the head-mounted display system, offers associated with behavioral characteristics of the wearer of the head-mounted display system, or any combination thereof.

[0160] In aspect 18, a head-mounted display system according to any one of aspects 1 to 17, wherein the display is configured to display a consent request to a wearer of the head-mounted display, and wherein in order to convey the consent request to the wearer, the hardware processor is configured to display the consent request to the wearer using the display.

[0161] In a 19th aspect, a head mounted display system according to any one of aspects 1 to 18, wherein the display is configured to present multiple depth planes to the wearer.

[0162] In a 20th aspect, a head mounted display system according to any one of aspects 1 to 19, wherein the display is configured to present a light field image to the wearer.

[0163] In aspect 21, a head mounted display system according to any one of aspects 1 to 20, wherein the display comprises a plurality of stacked waveguides.

[0164] In a 22nd aspect, a method for acquiring biometric information is disclosed. The method is under the control of a processor and includes: transmitting a consent request to a user, the request including a plurality of consent categories for the user's biometric information, the biometric information including physiological information, behavioral information, or any combination thereof; receiving a response to the consent request; determining that the response includes a consent indication from the user regarding the consent category that the user has indicated as agreeing to; obtaining a plurality of eye images of the user; and in response to a request for biometric information from a biometric information acquirer computing device, transmitting the plurality of eye images to the biometric information acquirer computing device based at least in part on the consent categories that the user has indicated as agreeing to and the request for biometric information.

[0165] In a 23rd aspect, the method of aspect 22, wherein the plurality of consent categories for the user's biometric information includes at least one of a quote category, a partial share category, and a full share category.

[0166] In aspect 24, the method according to aspect 23, wherein the consent category to which the user has indicated assent corresponds to a partial sharing category.

[0167] In aspect 25, the method of any of aspects 23 to 24, wherein the partial sharing category comprises sharing of a plurality of eye images on a sign associated with an item from the electronic catalog.

[0168] In aspect 26, the method according to any one of aspects 22 to 25 further includes: performing biometric data operations on multiple eye images to obtain biometric information; and in response to a request for biometric information from a biometric information acquirer computing device, transmitting a portion of the biometric information based at least in part on a consent category that the user has indicated agreement with.

[0169] In aspect 27, the method of aspect 26, wherein the biometric information corresponds to at least one of eye patterns, eye movements, eye gaze, or eye browsing history.

[0170] In aspect 28, the method of any one of aspects 22 to 27, wherein performing biometric data operations on the plurality of eye images to obtain biometric information comprises performing pose estimation or iris code generation on the plurality of eye images.

[0171] In aspect 29, the method according to any one of aspects 22 to 28 further comprises: displaying the consent request using a head mounted display.

[0172] In aspect 30, the method according to any one of aspects 22 to 29 further includes: in response to receiving a response to the consent request, displaying multiple biometric exchange offers using a head-mounted display, wherein the multiple biometric exchange offers include offers associated with biometric information about items from an electronic catalog or offers associated with biometric information based on a user's location.

[0173] In a 31st aspect, the method according to any one of aspects 22 to 30 further comprises: filtering the plurality of biometric exchange offers based on a filtering condition.

[0174] In aspect 32, the method of any one of aspects 22 to 31 further comprises receiving an external world image from a second imaging system coupled to the head mounted display.

[0175] In a 33rd aspect, the method according to aspect 32 further comprises processing a plurality of external world images captured by the plurality of eye images and the second image to obtain behavioral characteristics about the user.

[0176] In aspect 34, the method of aspect 33, wherein the behavioral characteristic about the user includes the user viewing a sign associated with an item from the electronic catalog.

[0177] In a 35th aspect, a method for exchanging biometric information is disclosed. The method is under the control of a processor and includes: causing a consent request regarding a user's biometric information to be displayed; determining that a response to the consent request includes a consent indication regarding an aspect of the biometric information; obtaining a biometric information processing result based on the biometric information processing; and providing the biometric information processing result to a biometric information acquirer computing device associated with the consent indication.

[0178] In a 36th aspect, the method of aspect 35, wherein the consent request comprises a request to provide a quote regarding biometric information.

[0179] In aspect 37, a method according to any one of aspects 35 to 36, wherein the consent request includes a plurality of consent categories regarding the biometric information.

[0180] In aspect 38, a method according to any one of aspects 35 to 37, wherein the request for consent includes a request to share any biometric information.

[0181] In aspect 39, the method according to any one of aspects 35 to 38 further comprises: receiving a request for biometric information from a biometric information acquirer computing device.

[0182] In aspect 40, the method of any one of aspects 35 to 39, wherein determining that the response to the consent request includes a consent indication regarding an aspect of the biometric information comprises determining that the consent indication includes an approval of an aspect of sharing the biometric information.

[0183] In aspect 41, the method according to any one of aspects 35 to 40 further comprises: receiving a consent request regarding the biometric information from the biometric information acquirer computing device.

[0184] In aspect 42, the method of any one of aspects 35 to 41, wherein determining that a response to a consent request includes a consent indication regarding aspects of biometric information comprises determining that a consent indication has been received from an optional item on a displayed page, wherein the optional item is associated with text content of a portion including the consent request.

[0185] In a 43rd aspect, a method for exchanging biometric information is disclosed. The method is under the control of a processor and includes: receiving an indication of consent by a wearer of a head mounted display to exchange a class of biometric information obtainable from the wearer of the head mounted display with a biometric information acquirer; obtaining biometric information of the wearer of the head mounted display; and transmitting biometric exchange data associated with the obtained biometric information to the biometric information acquirer.

[0186] In aspect 44, the method of aspect 43, wherein the biometric information comprises physiological information, behavioral information, or any combination thereof.

[0187] In a 45th aspect, the method according to any one of aspects 43 to 44 further comprises providing compensation to the wearer of the head mounted display or a designee of the wearer of the head mounted display.

[0188] In a 46th aspect, the method according to any one of aspects 43 to 45 further comprises receiving payment from the biometric information acquirer or a designee of the biometric information acquirer.

[0189] In a 47th aspect, the method according to any one of aspects 43 to 46, further comprising processing the obtained biometric information to determine biometric exchange data.

[0190] In aspect 48, the method of aspect 47, wherein processing the obtained biometric information comprises receiving processing instructions from a biometric information acquirer.

[0191] in conclusion

[0192] Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying drawings may be embodied in and fully or partially automated by: code modules executed by one or more physical computing systems, hardware computer processors, special-purpose circuits; and / or electronic hardware configured to execute specific and particular computer instructions. For example, a computing system can include a general-purpose computer (e.g., a server), a special-purpose circuit, etc., programmed with specific computer instructions or a special-purpose computer. Code modules may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpretive programming language. In some embodiments, specific operations and methods may be performed by circuits specific to a given function.

[0193] Furthermore, certain implementations of the functionality of the present disclosure are mathematically, computationally, or technically complex enough that dedicated hardware or one or more physical computing devices (with appropriate dedicated executable instructions) may be necessary in order to perform the functionality (e.g., due to the amount or complexity of the computations involved) or to provide results in substantially real time. For example, a video may include many frames, each with millions of pixels, and programmed computer hardware may be necessary in order to process the video data to provide a desired image processing task or application in a commercially reasonable amount of time.

[0194] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer memory, including hard drives, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disks, volatile or non-volatile memory, and combinations of the same and / or similar elements. Methods and modules (or data) may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media, including wireless-based media and wire / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital data packets or frames). The results of the disclosed processes or processing steps may be stored persistently or otherwise in any type of non-transitory physical computer memory, or may be transmitted via a computer-readable transmission medium.

[0195] Any process, frame, state, step or function in the flowchart described herein and / or depicted in the accompanying drawings should be understood as potentially representing a code module, code segment or code portion, which includes one or more executable instructions that implement a specific function (such as a logical function or an arithmetic function) or step in the process. Various processes, frames, states, steps or functions can be combined, rearranged, added, deleted, modified or otherwise changed according to the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any specific order, and the frames, steps or states associated therewith can be executed in other appropriate orders, such as in serial, parallel or some other manner. Tasks or events can be added to or removed from the disclosed example embodiments. In addition, the separation of various system components in the embodiments described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variants are feasible.

[0196] The processes, methods and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowd-sourced computing networks, the Internet and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0197] The systems and methods of the present disclosure each have several innovative aspects, any single aspect of which is not solely responsible for the desired properties disclosed herein or is not necessary for the desired properties disclosed herein. The various features and processes described above may be used independently of each other or may be used in combination in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Various modifications to the embodiments described in this disclosure may be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the spirit or scope of this disclosure. Therefore, the claims are not intended to be limited to the embodiments shown herein, but should be given the widest scope consistent with this disclosure, principles, and novel features disclosed herein.

[0198] Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as being performed in certain combinations, or even initially claimed as such, in some cases, one or more features from a claimed combination can be deleted from that combination, and a claimed combination may involve a sub-combination or a variation of a sub-combination. No single feature or group of features is necessary or integral to every embodiment.

[0199] Conditional terms used herein, such as (among other things) "can", "can", "may", "may", "can", "for example", etc., are generally intended to express that certain embodiments include and other embodiments do not include certain features, elements and / or steps, unless otherwise specifically stated or otherwise understood in the context. Therefore, such conditional terms are generally not intended to imply that features, elements and / or steps are necessary for one or more embodiments in any way, or that one or more embodiments necessarily include logic for determining whether these features, elements and / or steps are included in or will be performed in any specific embodiment with or without programmer input or prompting. The terms "include", "comprise", "have", etc. are synonyms and are used inclusively in an open-ended manner, and do not exclude additional elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusive meaning (rather than its exclusive meaning), so when used, for example, to connect a list of elements, the term "or" means one, some or all of the elements in the list. In addition, unless otherwise specifically stated, the articles "one", "an", and "said" used in this application and the appended claims should be interpreted as meaning "one or more" or "at least one".

[0200] As used herein, a phrase referring to "at least one" of a list of items refers to any combination of those items, including single members. As an example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B and C. Conjunctions such as the phrase "at least one of X, Y, and Z" (unless otherwise stated) are to be understood in the context of common usage to express that an item, term, etc. may be at least one of X, Y, or Z. Thus, such conjunctions are generally not intended to imply that certain embodiments require that each of at least one of X, at least one of Y, and at least one of Z be present.

[0201] Similarly, although the operations may be depicted in a particular order in the accompanying drawings, it should be appreciated that such operations do not need to be performed in the particular order or in a sequential order, or all the illustrated operations need to be performed to achieve the desired result. In addition, the accompanying drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations not shown can be incorporated into the schematically illustrated example methods and processes. For example, one or more additional operations can be performed before, after, at the same time, or during any of the illustrated operations. In addition, in other embodiments, the operations may be rearranged or reordered. In some cases, multitasking and parallel processing may have advantages. In addition, the separation of various system components described in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems can generally be integrated together in a single software product or packaged into multiple software products. In addition, other embodiments are within the scope of the following claims. In some cases, the actions listed in the claims can be performed in different orders and still achieve the desired result. Depicted in a particular order, it should be appreciated that such operations do not need to be performed in the particular order or in a sequential order, or all the illustrated operations need to be performed to achieve the desired result. In addition, the accompanying drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations not shown can be incorporated into the schematically shown example methods and processes. For example, one or more additional operations can be performed before, after, simultaneously or during any of the shown operations. In addition, in other embodiments, the operations can be rearranged or reordered. In some cases, multitasking and parallel processing can have advantages. In addition, the separation of various system components described in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems can generally be integrated together in a single software product or packaged into multiple software products. In addition, other embodiments are within the scope of the following claims. In some cases, the actions listed in the claims can be performed in different orders and still achieve the desired results.

Claims

1. A method for exchanging biometric information, comprising: determining biometric information of a wearer of the head mounted display based at least in part on the captured eye image data, wherein the biometric information is associated with a physiological characteristic or a behavioral characteristic of the wearer, and wherein determining the biometric information comprises segmenting one or more regions of at least one eye image in the captured eye image data that correspond to an iris of at least one eye of the wearer; electronically accessing biometric data sharing preferences associated with the wearer, wherein the biometric data sharing preferences include a first sharing preference category and a second sharing preference category; detecting initiation of a first action performed by the wearer based at least in part on the determined biometric information, wherein the first action is defined by a biometric information acquirer, detecting initiation of the first action based at least in part on determining that the at least one eye of the wearer is pointing in a particular direction, the particular direction determined based on analyzing one or more regions of the at least one eye image in the segmented captured eye image data; detecting termination of the first action based at least in part on the determined biometric information, wherein performance of the first action occurs for a length of time beginning at a detected first time of initiation of the first action and ending at a detected second time of termination of the first action; responsive to determining that the biometric data sharing preferences indicate that the wearer does not subscribe to the first shared preference category, preventing transmission of biometric information associated with the first shared preference category to a biometric information acquirer; as well as In response to determining that the biometric data sharing preferences indicate that the wearer has subscribed to the second shared preference category, transmitting biometric information to the biometric information acquirer, the biometric information (i) being associated with the second shared preference category and (ii) being determined during performance of the first action.

2. The method according to claim 1, wherein: The biometric data sharing preferences include data associated with at least one of a bid category, a partial share category, or a full share category.

3. The method according to claim 2, wherein: The partial sharing category includes sharing of multiple eye images regarding signage associated with items from an electronic catalog.

4. The method according to claim 3, further comprising: In response to receiving an indication of an adjustment to the adjustable biometric data sharing preference, display instructions configured to facilitate display of a biometric exchange offer using a display associated with the head mounted display system are generated.

5. The method according to claim 4, wherein: The biometric exchange offer includes an offer associated with biometric information about an item from an electronic catalog or an offer associated with biometric information based on the wearer's location.

6. The method according to claim 5, further comprising: The biometric exchange offers are filtered or sorted based on a threshold exchange price.

7. The method according to claim 5, further comprising: The biometric exchange offers are filtered or sorted based on exclusion of aspects of the biometric information.

8. The method according to claim 1, further comprising: A secure connection is formed between the head mounted display system and a computing device associated with the biometric information acquirer.

9. The method according to claim 1, wherein: The performance of the first action is based at least in part on one or more of a preconfigured time, a preconfigured time period, a location, or a plurality of locations.

10. The method according to claim 1, wherein: Transmission of biometric information to the biometric information acquirer is performed while also preventing transmission of biometric information to the biometric information acquirer that is (i) associated with the second shared preference category and (ii) determined before initiation or after termination of the first action.

11. A computer system comprising: a computer-readable medium storing computer-executable instructions; as well as One or more processors in communication with the computer-readable medium and configured to execute the computer-executable instructions to cause the computer system to perform a method according to any one of claims 1 to 10.

12. A method for exchanging biometric information, comprising: Acquire an eye image of a wearer of a head mounted display system, the eye image being captured by an image capture device of the head mounted display system; determining biometric information of the wearer based at least in part on the eye images, wherein the biometric information is associated with a physiological characteristic or a behavioral characteristic of the wearer, and wherein determining the biometric information comprises segmenting one or more regions of at least one of the eye images of the wearer corresponding to an iris of at least one eye of the wearer; transmitting, using a display and user interface of the head mounted display system, the transmission comprising a consent request to the wearer, the consent request comprising a plurality of consent categories for biometric information, wherein the plurality of consent categories comprises a first consent category and a second consent category; receiving a response to the consent request, wherein the response includes an indication of consent from the wearer, wherein the indication of consent from the wearer is based at least in part on the determined biometric information of the wearer, and wherein the indication of consent is based at least in part on determining that the at least one eye of the wearer is pointing in a particular direction, the particular direction determined based on analyzing one or more regions of the at least one eye image in the segmented eye images; In response to determining that the consent indication indicates that the wearer does not agree to the first consent category, and in response to a first request for biometric information, preventing the transmission of biometric information, the biometric information (1) being associated with the first consent category and (2) being determined based on an eye image obtained during execution of the consent indication; as well as In response to determining that the consent indication indicates that the wearer has agreed to the second consent category, and in response to a second request for biometric information, transmitting the biometric information to a device acquirer associated with the second request, the biometric information (1) being associated with the second consent category and (2) being determined based on an eye image obtained during execution of the consent indication.

13. The method according to claim 12, wherein: The plurality of consent categories include at least one of a quotation category, a partial sharing category, a full sharing category, or any combination thereof.

14. The method according to claim 13, wherein: The second consent category corresponds to the full sharing category or the partial sharing category.

15. The method according to claim 13, wherein: The partial sharing category includes sharing of multiple eye images regarding signage associated with items from an electronic catalog.

16. The method according to claim 13, wherein: The partial sharing category includes sharing of the biometric information regarding an item brand or an item product in a retail store.

17. The method according to claim 12, wherein: In response to the second request for biometric information from the device acquirer: The portion of the biometric information obtained based at least in part on the second consent category is transmitted.

18. The method according to claim 17, wherein: The biometric information corresponds to at least one of eye patterns, eye movements, eye gazes, or eye browsing history.

19. The method according to claim 18, further comprising: To determine the biometric information, pose estimation or iris code generation is performed using the eye image.

20. The method of claim 12, further comprising: The consent request is displayed using the head mounted display system.

21. The method according to any one of claims 12 to 20, further comprising: In response to receiving the consent indication, displaying a plurality of biometric exchange offers using the head mounted display system, Wherein the plurality of biometric exchange offers include offers associated with biometric information regarding items from an electronic catalog or offers associated with biometric information based on a location of the wearer.

22. The method according to claim 21, further comprising: The plurality of biometric exchange offers are filtered based on a threshold exchange price.

23. The method according to claim 21, further comprising: A plurality of biometric information exchange offers are filtered based on exclusions of aspects of biometric information, wherein the exclusions are specified by the wearer.

24. The method of claim 21, further comprising: The plurality of biometric exchange offers are filtered based on inclusion of aspects of the offers, wherein the inclusion is specified by the wearer.

25. The method of claim 12, further comprising: A secure connection is formed between the head mounted display system and the device acquirer.

26. The method of claim 12, further comprising: The eye images captured by the image capture device are processed to obtain behavioral characteristics about the wearer.

27. The method according to claim 26, wherein: The behavioral characteristics about the wearer include, in the eye image determined using a plurality of external world images, viewing of an item brand or an item product in a retail store by the wearer's eyes.

28. The method of claim 26, further comprising: A second consent request regarding the obtained behavioral characteristics is displayed using the head mounted display system.

29. The method according to claim 28, further comprising: in response to receiving a response to the second consent request regarding the obtained behavioral characteristic, displaying a second plurality of biometric exchange offers using the head mounted display system, wherein the second plurality of biometric exchange offers comprises: a quote associated with biometric information about an item from an electronic catalog, an offer associated with biometric information based on a location of the wearer of the head mounted display system, a quote associated with a behavioral characteristic of the wearer of the head mounted display system, or Any combination thereof.

30. The method of claim 12, wherein: Transmitting the consent request to the wearer includes displaying the consent request to the wearer using the head mounted display system.

31. The method of claim 12, wherein: The head mounted display system is configured to present multiple depth planes to the wearer.

32. The method of claim 12, wherein: The head-mounted display system is configured to present a light field image to the wearer.

33. The method of claim 12, wherein: The head mounted display system includes a plurality of stacked waveguides.

Images