Eye center-of-rotation determination, depth plane selection, and render camera positioning in display systems

JP2025023960AInactive Publication Date: 2025-02-19MAGIC LEAP INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024189674
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2018-07-24
Filing Date
2024-10-29
Publication Date
2025-02-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Benefits of technology

【0369】 本明細書に説明される主題の1つ以上の実装の詳細が、付随の図面および以下の説明に記載される。他の特徴、側面、および利点は、説明、図面、および請求項から明白となるであろう。本概要または以下の発明を実施するための形態のいずれも、本発明の主題の範囲を定義または限定することを主張するものではない。 本発明は、例えば、以下を提供する。 (項目1) ディスプレイシステムであって、前記ディスプレイシステムは、光をユーザの眼に投影し、仮想画像コンテンツを前記ユーザの視野内に表示するように構成され、前記眼は、角膜と、虹彩と、瞳孔と、水晶体と、網膜と、光学軸とを有し、前記光学軸は、前記水晶体、瞳孔、および角膜を通して延在し、前記ディスプレイシステムは、 前記ユーザの頭部上に支持されるように構成されるフレームと、 前記フレーム上に配置される頭部搭載型ディスプレイであって、前記ディスプレイは、異なる量の発散およびコリメーションのうちの少なくとも1つにおいて、光を前記ユーザの眼の中に投影し、仮想画像コンテンツを前記ユーザの視野に表示するように構成され、これにより、前記表示される仮想画像コンテンツは、異なる深度から異なる期間において生じるように現れる、頭部搭載型ディスプレイと、 前記ユーザの眼を結像するように構成される1つ以上の眼追跡カメラと、 前記ディスプレイおよび前記1つ以上の眼追跡カメラと通信する処理電子機器であって、前記処理電子機器は、前記1つ以上の眼追跡カメラを用いて取得された前記眼の画像に基づいて、前記眼の回転中心の推定値を取得するように構成される、処理電子機器と を備える、ディスプレイシステム。 (項目2) 1つ以上の光源をさらに備え、前記1つ以上の光源は、前記ユーザの眼に対して前記フレーム上に配置され、前記ユーザの眼を照明し、前記1つ以上の眼追跡カメラは、前記1つ以上の光源からの光を使用して、前記眼の画像を形成する、項目1に記載のディスプレイシステム。 (項目3) 前記1つ以上の光源は、少なくとも2つの光源を備え、前記少なくとも2つの光源は、前記ユーザの眼に対して前記フレーム上に配置され、前記ユーザの眼を照明する、項目2に記載のディスプレイシステム。 (項目4) 前記1つ以上の光源は、赤外線光エミッタを備える、項目3に記載のディスプレイシステム。 (項目5) 前記1つ以上の光源は、1つ以上の閃光を前記眼上に形成し、前記処理電子機器は、前記1つ以上の閃光に基づいて、前記角膜の場所を決定するように構成される、項目2に記載のディスプレイシステム。 (項目6) 前記角膜は、それと関連付けられた角膜球体を有し、前記角膜球体は、曲率中心を有し、前記処理電子機器は、前記1つ以上の閃光に基づいて、前記角膜球体の曲率中心の場所を決定するように構成される、項目5に記載のディスプレイシステム。 (項目7) 前記角膜は、それと関連付けられた角膜球体を有し、前記角膜球体は、曲率中心を有し、前記処理電子機器は、前記角膜球体の曲率中心の場所を決定するように構成される、項目1に記載のディスプレイシステム。 (項目8) 前記1つ以上の眼追跡カメラは、前記眼の瞳孔を結像するように構成される、項目1に記載のディスプレイシステム。 (項目9) 前記処理電子機器は、前記瞳孔の中心の場所を決定するように構成される、項目1に記載のディスプレイシステム。 (項目10) 前記処理電子機器は、前記虹彩と前記瞳孔との間の境界の少なくとも一部を決定するように構成される、項目1に記載のディスプレイシステム。 (項目11) 前記処理電子機器は、前記虹彩と前記瞳孔との間の境界の中心を決定するように構成される、項目10に記載のディスプレイシステム。 (項目12) 前記処理電子機器は、前記角膜の曲率中心に対する3次元空間内の前記瞳孔の中心の場所を決定するように構成される、項目1に記載のディスプレイシステム。 (項目13) 前記処理電子機器は、3次元空間内の前記瞳孔の中心の場所に基づいて、前記光学軸の場所および配向を決定するように構成される、項目12に記載のディスプレイシステム。(項目14) 前記処理電子機器は、前記光学軸の場所および配向を決定するように構成される、項目1に記載のディスプレイシステム。 (項目15) 前記処理電子機器は、前記角膜の曲率中心に対する3次元空間内の前記瞳孔の中心の場所に基づいて、前記光学軸の場所および配向を決定するように構成される、項目14に記載のディスプレイシステム。 (項目16) 前記処理電子機器は、前記角膜の曲率中心に基づいて、前記眼の回転中心の場所を決定するように構成される、項目1に記載のディスプレイシステム。 (項目17) 前記処理電子機器は、前記角膜の曲率中心および前記光学軸の場所および配向に基づいて、前記眼の回転中心の場所を決定するように構成される、項目1に記載のディスプレイシステム。 (項目18) 前記処理電子機器は、前記角膜の曲率中心から前記光学軸に沿った特定の距離を平行移動させることによって、前記眼の回転中心の場所を決定するように構成される、項目17に記載のディスプレイシステム。 (項目19) 前記曲率中心から前記回転中心までの特定の距離は、4.0mm~6.0mmである、項目18に記載のディスプレイシステム。 (項目20) 前記曲率中心から前記回転中心までの特定の距離は、約4.7mmである、項目18に記載のディスプレイシステム。 (項目21) 前記特定の距離は、固定される、項目18に記載のディスプレイシステム。 (項目22) 前記処理電子機器は、少なくとも、前記1つ以上の眼追跡カメラを用いて以前に取得された前記眼の1つ以上の画像に基づいて、前記特定の距離を決定するように構成される、項目18に記載のディスプレイシステム。 (項目23) 前記処理電子機器は、前記光学軸の場所および配向に基づいて、前記光学軸からオフセットされた視軸の場所および配向を決定するように構成される、項目1に記載のディスプレイシステム。 (項目24) 前記処理電子機器は、前記光学軸に対する角回転に基づいて、視軸の場所および配向を決定するように構成される、項目1に記載のディスプレイシステム。 (項目25) 前記処理電子機器は、前記光学軸に対する4.0°~6.5°の角回転に基づいて、視軸の場所および配向を決定するように構成される、項目1に記載のディスプレイシステム。 (項目26) 前記処理電子機器は、前記光学軸に対する約5.2°の角回転に基づいて、視軸の場所および配向を決定するように構成される、項目1に記載のディスプレイシステム。 (項目27) 前記処理電子機器は、少なくとも、前記1つ以上の眼追跡カメラを用いて以前に取得された前記眼の1つ以上の画像に基づいて、視軸の場所および配向を決定するように構成される、項目1に記載のディスプレイシステム。 (項目28) 前記処理電子機器は、前記眼が回転している期間にわたって、前記光学軸または視軸の場所の複数の決定に基づいて、前記眼の回転中心を決定するように構成される、項目1に記載のディスプレイシステム。 (項目29) 前記処理電子機器は、前記眼が回転している期間にわたって、前記光学軸または視軸の場所の複数の決定の交差の領域を識別することによって、前記眼の回転中心を決定するように構成される、項目1に記載のディスプレイシステム。 (項目30) 前記処理電子機器は、前記ユーザの左および右眼に関する光学軸の場所および配向の決定に基づいて、前記ユーザの左および右眼が注視している前記ユーザの輻輳·開散運動距離を決定するように構成される、項目1に記載のディスプレイシステム。 (項目31) 前記処理電子機器は、前記ユーザの左および右眼に関する視軸の場所および配向の決定に基づいて、前記ユーザの左および右眼が注視している前記ユーザの輻輳·開散運動距離を決定するように構成される、項目1に記載のディスプレイシステム。 (項目32) 前記処理電子機器は、前記ユーザの左および右眼に関する視軸の交差の領域を識別することに基づいて、前記ユーザの左および右眼が注視している輻輳·開散運動距離を決定するように構成される、項目1に記載のディスプレイシステム。 (項目33) 前記処理電子機器は、前記左および右眼に関する視軸を水平平面上に投影し、前記水平平面上における前記左および右眼に関する視軸の投影の交差の領域を識別することによって、前記ユーザの左および右眼が注視している輻輳·開散運動距離を決定するように構成される、項目1に記載のディスプレイシステム。 (項目34) 前記処理電子機器は、前記ユーザの輻輳·開散運動距離の決定に基づいて、画像コンテンツを投影するための発散およびコリメーションのうちの少なくとも1つの相対的量を決定するように構成される、項目1に記載のディスプレイシステム。 (項目35) 前記処理電子機器は、前記フレーム上の電子機器を含む、項目1に記載のディスプレイシステム。 (項目36) 前記処理電子機器は、前記フレーム上の電子機器と、前記フレームから遠隔の場所に配置される電子機器とを含む、項目1に記載のディスプレイシステム。 (項目37) 前記処理電子機器は、前記フレーム上の電子機器と、ベルトパック上の電子機器とを含む、項目1に記載のディスプレイシステム。 (項目38) 前記ディスプレイの少なくとも一部は、透明であり、前記ユーザが前記頭部搭載型ディスプレイを装着すると、前記透明部分が、前記ユーザおよび前記頭部搭載型ディスプレイの正面の環境の一部からの光をユーザの眼に透過させ、前記ユーザおよび前記頭部搭載型ディスプレイの正面の環境の一部のビューを提供するように、前記ユーザの眼の正面の場所に配置される、項目1に記載のディスプレイシステム。 (項目39) 前記頭部搭載型ディスプレイは、第1の発散量において、前記ユーザの正面の環境の一部からの光を受け取り、前記第1の発散量に実質的に類似する第2の発散量を伴って、前記ユーザの正面の環境の一部からの光をユーザの眼に透過させる、項目1に記載のディスプレイシステム。 (項目40) 前記処理電子機器は、複数の推定される回転中心位置をフィルタリングする、平均する、カルマンフィルタを適用する、またはそれらの任意の組み合わせによって、前記回転中心の推定値を取得するように構成される、項目1に記載のディスプレイシステム。 (項目41) 前記処理電子機器は、前記ユーザの眼の回転中心の決定された位置に開口を有するカメラによって捕捉されたかのようにレンダリングされる前記仮想画像コンテンツを前記ユーザの眼に提示するように構成される、項目1に記載のディスプレイシステム。 (項目42) 前記処理電子機器は、レンダリングカメラを前記回転中心において使用し、仮想画像を前記眼に提示されるようにレンダリングするように構成される、項目1に記載のディスプレイシステム。 (項目43) 前記処理電子機器は、前記眼の網膜より前記回転中心に近い開口を有するカメラによって捕捉されたかのようにレンダリングされる仮想画像を前記眼に提示されるようにレンダリングするように構成されるレンダリングカメラを使用するように構成される、項目1に記載のディスプレイシステム。 (項目44) 前記処理電子機器は、前記眼の回転中心に開口を有するカメラによって捕捉されたかのようにレンダリングされる仮想画像を前記眼に提示されるようにレンダリングするように構成されるレンダリングカメラを使用するように構成される、項目1に記載のディスプレイシステム。 (項目45) 前記処理電子機器は、レンダリングカメラを前記回転中心において使用し、仮想画像を前記眼に提示されるようにレンダリングするように構成され、前記レンダリングカメラは、前記眼の回転中心に開口を伴ってモデル化される、項目1に記載のディスプレイシステム。 (項目46) 前記処理電子機器は、前記1つ以上の眼追跡カメラによって取得された前記眼の画像に基づいて、前記眼が回転している期間にわたって、前記ユーザの眼の複数の視線方向の決定に基づいて、前記眼の回転中心の推定値を取得するように構成される、項目1に記載のディスプレイシステム。 (項目47) 前記処理電子機器は、前記眼が回転している期間にわたって、前記1つ以上の眼追跡カメラを用いて取得された画像内の前記ユーザの眼の瞳孔、虹彩、または角膜輪部のうちの1つ以上のものの形状の変動量に基づいて、前記視線方向を決定するように構成される、項目46に記載のディスプレイシステム。 (項目48) 前記処理電子機器は、前記1つ以上の眼追跡カメラを用いて取得された前記ユーザの眼の画像上の複数の空間場所に基づいて、位置のアレイを決定するように構成される、項目1に記載のディスプレイシステム。 (項目49) 前記位置のアレイは、楕円の少なくとも一部に対応する、項目48に記載のディスプレイシステム。 (項目50) 前記処理電子機器は、曲線を前記ユーザの眼の画像上の複数の空間場所に適合させることによって、前記位置のアレイを決定するように構成される、項目48に記載のディスプレイシステム。 (項目51) 前記曲線は、楕円を備える、項目50に記載のディスプレイシステム。 (項目52) 前記画像上の複数の空間場所は、前記画像内の前記ユーザの眼の前記角膜輪部上の空間場所を備える、項目49に記載のディスプレイシステム。 (項目53) 前記画像上の複数の空間場所は、前記画像内の前記ユーザの眼の前記虹彩と強膜との間の境界上の空間場所を備える、項目49に記載のディスプレイシステム。 (項目54) 前記画像上の複数の空間場所は、前記1つ以上の眼追跡カメラを用いて取得された前記画像内の前記ユーザの眼の前記角膜と前記強膜との間の境界上の空間場所を備える、項目49に記載のディスプレイシステム。 (項目55) 前記処理電子機器は、前記位置のアレイの第1の側上の場所から前記位置のアレイを通して前記位置のアレイの第2の反対側まで延在する複数の線形経路を決定するように構成される、項目49に記載のディスプレイシステム。 (項目56) 前記処理電子機器は、前記複数の線形経路に基づいて、円形領域を決定するように構成され、前記円形領域は、半径Rを有する、項目55に記載のディスプレイシステム。 (項目57) 前記半径Rは、角膜輪部の平均半径に対応する、項目56に記載のディスプレイシステム。 (項目58) 前記半径Rは、前記ユーザの眼の角膜輪部の測定された半径に対応する、項目56に記載のディスプレイシステム。 (項目59) 前記半径Rは、瞳孔の平均半径に対応する、項目56に記載のディスプレイシステム。(項目60) 前記半径Rは、前記ユーザの眼の瞳孔の測定された半径に対応する、項目56に記載のディスプレイシステム。 (項目61) 前記処理電子機器は、前記円形領域の中心部分を通る法線の場所および方向を決定するように構成される、項目56に記載のディスプレイシステム。 (項目62) 前記処理電子機器は、前記1つ以上の眼追跡カメラを用いて以前に取得された前記眼の複数の画像に基づいて、個別の円形領域の中心部分を通る複数の法線の個別の場所および方向を決定するように構成される、項目56に記載のディスプレイシステム。 (項目63) 前記処理電子機器は、前記複数の法線が収束または交差する位置を決定するように構成される、項目62に記載のディスプレイシステム。 (項目64) 前記処理電子機器は、前記眼が回転している期間にわたって取得された前記ユーザの眼の画像に基づいて決定される前記法線の複数のものの交差の領域を識別することによって、前記ユーザの眼の回転中心の推定値を取得するように構成される、項目62に記載のディスプレイシステム。 (項目65) 前記処理電子機器は、前記眼が回転している期間にわたって取得された前記ユーザの眼の画像に基づいて決定される前記複数の法線の複数のものの場所および方向に基づいて、前記ユーザの眼の回転中心の推定値を取得するように構成される、項目62に記載のディスプレイシステム。 (項目66) 前記アレイ位置の前記第1の側上の場所は、前記1つ以上の眼追跡カメラのうちの1つの座標系の原点に対応する、項目55に記載のディスプレイシステム。 (項目67) ディスプレイシステムであって、前記ディスプレイシステムは、光をユーザの眼に投影し、仮想画像コンテンツを前記ユーザの視野内に表示するように構成され、前記眼は、角膜と、虹彩と、瞳孔と、水晶体と、網膜と、光学軸とを有し、前記光学軸は、前記水晶体、瞳孔、および角膜を通して延在し、前記ディスプレイシステムは、 前記ユーザの頭部上に支持されるように構成されるフレームと、 前記フレーム上に配置される頭部搭載型ディスプレイであって、前記ディスプレイは、異なる量の発散およびコリメーションのうちの少なくとも1つにおいて、光を前記ユーザの眼の中に投影し、仮想画像コンテンツを前記ユーザの視野に表示するように構成され、これにより、前記表示される仮想画像コンテンツは、異なる深度から異なる期間において生じるように現れる、頭部搭載型ディスプレイと、 前記ユーザの眼を結像するように構成される1つ以上の眼追跡カメラと、 前記ディスプレイおよび前記1つ以上の眼追跡カメラと通信する処理電子機器であって、前記処理電子機器は、前記1つ以上の眼追跡カメラを用いて取得された前記眼の画像に基づいて、前記眼の視点中心の位置推定値を取得するように構成され、前記視点中心は、前記眼の瞳孔の近位にあるかまたは前記角膜と前記眼の瞳孔との間にあると推定される、処理電子機器と を備え、 前記処理電子機器は、前記視点中心に位置するレンダリングカメラによってレンダリングされる前記仮想画像コンテンツを前記ユーザの眼に提示するように構成される、ディスプレイシステム。 (項目68) 光をユーザの眼に投影し、前記仮想画像コンテンツを前記ユーザの視野内に表示するように構成されるディスプレイシステムにおいて、仮想画像コンテンツをレンダリングする方法であって、前記眼は、角膜と、虹彩と、瞳孔と、水晶体と、網膜と、光学軸とを有し、前記光学軸は、前記水晶体、瞳孔、および角膜を通して延在し、前記方法は、 前記ユーザの眼を結像し、前記眼の移動を追跡するように構成される1つ以上の眼追跡カメラを用いて、前記眼の回転中心の位置を決定することと、 レンダリングエンジンを用いて、前記眼の回転中心におけるレンダリングカメラを用いて、仮想画像コンテンツをレンダリングすることであって、前記レンダリングカメラは、仮想画像を前記眼に提示されるようにレンダリングするように構成される、ことと、 頭部搭載型ディスプレイを用いて、前記仮想画像コンテンツが異なる深度から異なる期間において生じるように現れるように、異なる量の発散において、光を前記ユーザの眼の中に投影し、前記レンダリングされた仮想画像コンテンツをユーザの視野に表示することと を含む、方法。 (項目69) ディスプレイシステムであって、前記ディスプレイシステムは、光をユーザの眼に投影し、仮想画像コンテンツを前記ユーザの視野内に表示するように構成され、前記眼は、角膜と、虹彩と、瞳孔と、水晶体と、網膜と、光学軸とを有し、前記光学軸は、前記水晶体、瞳孔、および角膜を通して延在し、前記ディスプレイシステムは、 前記ユーザの頭部上に支持されるように構成されるフレームと、 前記フレーム上に配置される頭部搭載型ディスプレイであって、前記ディスプレイは、異なる量の発散において、光を前記ユーザの眼の中に投影し、仮想画像コンテンツをユーザの視野に表示するように構成され、これにより、前記表示される仮想画像コンテンツは、異なる深度から異なる期間において生じるように現れ、前記頭部搭載型ディスプレイは、第1の発散量を有する光を、第1の期間において、前記ユーザの眼の中に投影するように構成され、第2の発散量を有する光を、第2の期間において、前記ユーザの眼の中に投影するように構成され、前記第1の発散量は、前記第2の発散量と異なる、頭部搭載型ディスプレイと、 前記ユーザの眼を結像するように構成される1つ以上の眼追跡カメラと、 前記ディスプレイおよび前記1つ以上の眼追跡カメラと通信する処理電子機器であって、前記処理電子機器は、前記1つ以上の眼追跡カメラを用いて取得された前記眼の画像に基づいて、前記眼の回転中心の推定値を取得し、前記1つ以上の眼追跡カメラを用いて取得された前記眼の画像に基づいて、前記ユーザの輻輳·開散運動距離の推定値を取得し、前記ユーザの推定される輻輳·開散運動距離に基づいて、前記第1の発散量における前記ユーザの眼の中への光の投影から前記第2の発散量における前記ユーザの眼の中への光の投影に偏移させるように構成される、処理電子機器と を備える、ディスプレイシステム。 (項目70) ディスプレイシステムであって、前記ディスプレイシステムは、光をユーザの左および右眼に投影し、仮想画像コンテンツを前記ユーザの視野内に表示するように構成され、前記眼はそれぞれ、角膜と、虹彩と、瞳孔と、水晶体と、網膜と、光学軸とを有し、前記光学軸は、前記水晶体、瞳孔、および角膜を通して延在し、前記ディスプレイシステムは、 前記ユーザの頭部上に支持されるように構成されるフレームと、 前記フレーム上に配置される頭部搭載型ディスプレイであって、前記ディスプレイは、異なる量の発散およびコリメーションのうちの少なくとも1つにおいて、光を前記ユーザの左および右眼の中に投影し、仮想画像コンテンツをユーザの視野に表示するように構成され、これにより、前記表示される仮想画像コンテンツは、前記ユーザの左および右眼からの異なる距離から異なる期間において生じるように現れる、頭部搭載型ディスプレイと、 前記ユーザの左眼を結像するように構成される第1の眼追跡カメラと、 前記ユーザの右眼を結像するように構成される第2の眼追跡カメラと、 前記ディスプレイおよび前記第1および第2の眼追跡カメラと通信する処理電子機器であって、前記処理電子機器は、前記第1および第2の眼追跡カメラを用いて取得された前記左および右眼の画像に基づいて、前記ユーザの左眼と右眼との間の瞳孔間距離の推定値を取得するように構成される、処理電子機器と を備える、ディスプレイシステム。 (項目71) 光をユーザの左および右眼に投影し、前記仮想画像コンテンツを前記ユーザの視野内に表示するように構成されるディスプレイシステムにおいて、仮想画像コンテンツをレンダリングする方法であって、前記眼はそれぞれ、角膜と、虹彩と、瞳孔と、水晶体と、網膜と、光学軸とを有し、前記光学軸は、前記水晶体、瞳孔、および角膜を通して延在し、前記方法は、 前記ユーザの眼を結像し、前記眼の移動を追跡するように構成される1つ以上の眼追跡カメラを用いて、前記左眼の回転中心の位置および前記右眼の回転中心の位置を決定することと、 前記1つ以上の眼追跡カメラと通信する処理電子機器を用いて、前記左および右眼の回転中心の決定された位置に基づいて、前記ユーザの瞳孔間距離を推定することと、 前記1つ以上の眼追跡カメラを用いて、現在の左眼姿勢および現在の右眼姿勢を決定することと、 前記処理電子機器を用いて、前記推定される瞳孔間距離と前記決定された現在の左眼姿勢および前記決定された現在の右眼姿勢を比較することによって、前記ユーザの現在の輻輳·開散運動距離を推定することと を含む、方法。 (項目72) ディスプレイシステムであって、前記ディスプレイシステムは、光をユーザの眼に投影し、仮想画像コンテンツを前記ユーザの視野内に表示するように構成され、前記眼は、角膜と、虹彩と、瞳孔と、水晶体と、網膜と、光学軸とを有し、前記光学軸は、前記水晶体、瞳孔、および角膜を通して延在し、前記ディスプレイシステムは、 前記ユーザの頭部上に支持されるように構成されるフレームと、 前記フレーム上に配置される頭部搭載型ディスプレイであって、前記ディスプレイは、異なる量の発散およびコリメーションのうちの少なくとも1つにおいて、光を前記ユーザの眼の中に投影し、仮想画像コンテンツを前記ユーザの視野に表示するように構成され、これにより、前記表示される仮想画像コンテンツは、異なる深度から異なる期間において生じるように現れる、頭部搭載型ディスプレイと、 前記ユーザの眼を結像するように構成される1つ以上の眼追跡カメラと、 前記ディスプレイおよび前記1つ以上の眼追跡カメラと通信する処理電子機器であって、前記処理電子機器は、前記1つ以上の眼追跡カメラを用いて取得された前記眼の画像に基づいて、前記眼の回転中心の位置推定値を取得するように構成され、前記画像に基づいて、前記眼の光学軸の方向推定値を取得するように構成される、処理電子機器と を備え、 前記処理電子機器は、前記光学軸に沿って配置され、前記網膜からある方向に6.0mm~13.0mm離れるように、前記眼の回転中心の推定される位置から離間される開口を有するカメラによって捕捉されたかのようにレンダリングされる前記仮想画像コンテンツを前記ユーザの眼に提示するように構成される、ディスプレイシステム。 (項目73) ディスプレイシステムであって、前記ディスプレイシステムは、光をユーザの眼に投影し、仮想画像コンテンツを前記ユーザの視野内に表示するように構成され、前記眼は、角膜と、虹彩と、瞳孔と、水晶体と、網膜と、光学軸とを有し、前記光学軸は、前記水晶体、瞳孔、および角膜を通して延在し、前記ディスプレイシステムは、 前記ユーザの頭部上に支持されるように構成されるフレームと、 前記フレーム上に配置される頭部搭載型ディスプレイであって、前記ディスプレイは、異なる量の発散およびコリメーションのうちの少なくとも1つにおいて、光を前記ユーザの眼の中に投影し、仮想画像コンテンツを前記ユーザの視野に表示するように構成され、これにより、前記表示される仮想画像コンテンツは、異なる深度から異なる期間において生じるように現れる、頭部搭載型ディスプレイと、 前記ユーザの眼を結像するように構成される1つ以上の眼追跡カメラと、 前記ディスプレイおよび前記1つ以上の眼追跡カメラと通信する処理電子機器と を備え、 前記処理電子機器は、前記眼の瞳孔に位置するかまたは前記眼の前記瞳孔と前記角膜との間に位置するレンダリングカメラによってレンダリングされる前記仮想画像コンテンツを前記ユーザの眼に提示するように構成される、ディスプレイシステム。 (項目74) ディスプレイシステムであって、前記ディスプレイシステムは、光をユーザの眼に投影し、仮想画像コンテンツを前記ユーザの視野内に表示するように構成され、前記眼は、角膜と、虹彩と、瞳孔と、水晶体と、網膜と、光学軸とを有し、前記光学軸は、前記水晶体、瞳孔、および角膜を通して延在し、前記ディスプレイシステムは、 前記ユーザの頭部上に支持されるように構成されるフレームと、 前記フレーム上に配置される頭部搭載型ディスプレイであって、前記ディスプレイは、光を前記ユーザの眼の中に投影し、仮想画像コンテンツを前記ユーザの視野に表示するように構成され、前記ディスプレイの少なくとも一部は、透明であり、前記ユーザが前記フレームを装着すると、前記透明部分が、前記ユーザおよび前記頭部搭載型ディスプレイの正面の環境の一部からの光をユーザの眼に透過させ、前記ユーザおよび前記頭部搭載型ディスプレイの正面の環境の一部のビューを提供するように、前記ユーザの眼の正面の場所に配置される、頭部搭載型ディスプレイと、 前記ユーザの眼を結像するように構成される内向きに面した結像システムと、 前記内向きに面した結像システムと通信する処理電子機器であって、前記処理電子機器は、前記内向きに面した結像システムを用いて取得された前記眼の複数の画像に基づいて、前記眼の回転中心の推定値を取得するように構成され、前記処理電子機器は、回転中心の計算される値における変動量を決定し、前記変動量に基づいて、回転中心の統計的に決定された推定値を選択するように構成される、処理電子機器と を備える、ディスプレイシステム。 (項目75) 光をユーザの眼に投影し、前記仮想画像コンテンツを前記ユーザの視野内に表示するように構成されるディスプレイシステムにおいて、仮想画像コンテンツをレンダリングする方法であって、前記眼は、角膜と、虹彩と、瞳孔と、水晶体と、網膜と、光学軸とを有し、前記光学軸は、前記水晶体、瞳孔、および角膜を通して延在し、前記方法は、 前記ユーザの眼を結像するように構成される1つ以上の眼カメラを用いて、前記1つ以上のカメラを用いた前記眼の結像に基づいて、位置を決定することと、 レンダリングエンジンを用いて、前記決定された位置に基づく場所におけるレンダリングカメラを用いて仮想画像コンテンツをレンダリングすることであって、前記レンダリングカメラは、仮想画像を前記眼に提示されるようにレンダリングするように構成される、ことと、 頭部搭載型ディスプレイを用いて、光を前記ユーザの眼の中に投影し、前記レンダリングされた仮想画像コンテンツをユーザの視野に表示することと を含む、方法。 (項目76) ディスプレイシステムであって、前記ディスプレイシステムは、光をユーザの眼に投影し、仮想画像コンテンツを前記ユーザの視野内に表示するように構成され、前記眼は、角膜と、虹彩と、瞳孔と、水晶体と、網膜と、光学軸とを有し、前記光学軸は、前記水晶体、瞳孔、および角膜を通して延在し、前記ディスプレイシステムは、 前記ユーザの頭部上に支持されるように構成されるフレームと、 前記フレーム上に配置される頭部搭載型ディスプレイであって、前記ディスプレイは、光を前記ユーザの眼の中に投影し、仮想画像コンテンツを前記ユーザの視野に表示するように構成される、頭部搭載型ディスプレイと、 前記ユーザの眼を結像するように構成される1つ以上のカメラと、 前記ディスプレイおよび前記1つ以上のカメラと通信する処理電子機器であって、前記処理電子機器は、前記1つ以上のカメラを用いて取得された前記眼の画像に基づいて、前記眼の位置を取得するように構成される、処理電子機器と を備え、 前記処理電子機器は、前記決定された位置に基づく場所に位置するレンダリングカメラによってレンダリングされる前記仮想画像コンテンツを前記ユーザの眼に提示するように構成される、ディスプレイシステム。 (項目77) ディスプレイシステムであって、前記ディスプレイシステムは、光をユーザの眼に投影し、仮想画像コンテンツを前記ユーザの視野内に表示するように構成され、前記眼は、角膜と、虹彩と、瞳孔と、水晶体と、網膜とを有し、前記ディスプレイシステムは、 前記ユーザの頭部上に支持されるように構成されるフレームと、 前記フレーム上に配置される頭部搭載型ディスプレイであって、前記ディスプレイは、異なる量の発散およびコリメーションのうちの少なくとも1つにおいて、光を前記ユーザの眼の中に投影し、仮想画像コンテンツを前記ユーザの視野に表示するように構成され、これにより、前記表示される仮想画像コンテンツは、異なる深度から異なる期間において生じるように現れる、頭部搭載型ディスプレイと、 前記ユーザの眼を結像するように構成される1つ以上の眼追跡カメラと、 前記ディスプレイおよび前記1つ以上の眼追跡カメラと通信する処理電子機器であって、前記処理電子機器は、 前記1つ以上の眼追跡カメラを用いて取得された前記眼の画像に基づいて、前記ユーザの眼の水晶体、瞳孔、および角膜を通して延在する光学軸の位置および配向推定値を取得することと、 前記眼の光学軸に位置合わせされる、レンダリング空間内のある軸に沿った特定の場所を識別することと、 レンダリング空間内の前記特定の場所に位置する仮想レンダリングカメラによってレンダリングされるように、前記仮想画像コンテンツを前記ユーザの眼に提示することと を行うように構成される、処理電子機器と を備える、ディスプレイシステム。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2025023960000001_ABST
    Figure 2025023960000001_ABST
Patent Text Reader

Abstract

To provide display systems, virtual reality, and augmented reality imaging and visualization systems.SOLUTION: A display system can include a head-mounted display configured to project light to an eye of a user to display virtual image content at different amounts of divergence and collimation. The display system can include an inward-facing imaging system that images the user's eye, and processing electronics that are in communication with the inward-facing imaging system and configured to obtain an estimate of a center of rotation of the user's eye. The display system may render image content with a virtual render camera positioned at the determined position of the center of rotation of the eye.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 62 / 618,559, entitled "EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS," filed January 17, 2018. This application further claims priority to U.S. Provisional Patent Application No. 62 / 702,849, entitled "EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS," filed July 24, 2018. Each of the above applications is incorporated herein by reference in its entirety.

[0002] This application further incorporates by reference U.S. Patent Publication No. 2018 / 0018515, entitled "IRIS BOUNDARY ESTIMATION USING CORNEA CURVATURE," published January 18, 2018.

[0003] The present disclosure relates to display systems, virtual reality, and augmented reality imaging and visualization systems, and more particularly to depth plane selection based in part on a user's interpupillary distance. [Background technology]

[0004] Modern computing and display technologies have facilitated the development of systems for so-called “virtual reality,” “augmented reality,” or “mixed reality” experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or can be perceived as real. Virtual reality, or “VR,” scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual input. Augmented reality, or “AR,” scenarios typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the real world around the user. Mixed reality, or “MR,” relates to the merging of real and virtual worlds to create new environments in which physical and virtual objects coexist and interact in real time. Consequently, the human visual perception system is highly complex, making it challenging to produce VR, AR, or MR technologies that facilitate comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention [Means for solving the problem]

[0005] Various embodiments of depth plane selection in a mixed reality system are disclosed.

[0006] The display system can be configured to project light onto a user's eye and display virtual image content within the user's field of view. The user's eye may have a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea. The display system can include a frame configured to be supported on the user's head, a head-mounted display disposed on the frame, the display configured to project light onto the user's eye and display virtual image content within the user's field of view with one of different amounts of divergence and collimation, such that the displayed virtual image content appears to arise from different depths and at different time periods, one or more eye-tracking cameras configured to image the user's eye, and processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to obtain an estimate of the center of rotation of the eye based on images of the eye obtained using the one or more eye-tracking cameras.

[0007] Described herein are various embodiments of display systems, such as those listed below, that project light onto one or both eyes of a user and display virtual image content within the user's field of view.

[0008] Example 1: a display system configured to project light onto a user's eye and display virtual image content within the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea, the display system including a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project the light into the user's eyes with at least one of different amounts of divergence and collimation to display virtual image content in the user's field of view, such that the displayed virtual image content appears to arise from different depths and at different time periods; one or more eye-tracking cameras configured to image the eyes of a user; processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to obtain an estimate of a center of rotation of the eye based on images of the eye acquired using the one or more eye-tracking cameras; A display system comprising:

[0009] Example 2: The display system of Example 1 further comprises one or more light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes, wherein the one or more eye tracking cameras form images of the eyes using light from the one or more light sources.

[0010] Example 3: 3. The display system of any one of Examples 1 to 2, wherein the one or more light sources comprise at least two light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes.

[0011] Example 4: 4. The display system of any one of examples 1 to 3, wherein the one or more light sources comprise infrared light emitters.

[0012] Example 5: A display system described in any of Examples 1-4, wherein the one or more light sources form one or more flashes of light on the eye, and the processing electronics are configured to determine the location of the cornea based on the one or more flashes of light.

[0013] Example 6: A display system described in any of Examples 1-5, wherein the cornea has a corneal sphere associated therewith, the cornea having a center of curvature, and the processing electronics is configured to determine the location of the center of curvature of the corneal sphere.

[0014] Example 7: The display system of Example 5, wherein the cornea has a corneal sphere associated therewith, the cornea having a center of curvature, and the processing electronics is configured to determine the location of the center of curvature of the corneal sphere based on the one or more flashes of light.

[0015] Example 8: 10. The display system of any preceding example embodiment, wherein the one or more eye-tracking cameras are configured to image the pupils of the eyes.

[0016] Example 9: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a location of a center of the pupil.

[0017] Example 10: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine at least a portion of a boundary between the iris and the pupil.

[0018] Example 11: 11. The display system of example 10, wherein the processing electronics is configured to determine a center of a boundary between the iris and the pupil.

[0019] Example 12: The display system of any of the preceding examples, wherein the processing electronics is configured to determine the location of the center of the pupil in three-dimensional space relative to the center of curvature of the cornea.

[0020] Example 13: The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a location and orientation of the optical axis.

[0021] Example 14: 13. The display system of example 12, wherein the processing electronics is configured to determine the location and orientation of the optical axis based on the location of the center of the pupil in three-dimensional space.

[0022] Example 15: 10. The display system of any preceding example, wherein the processing electronics is configured to determine the location and orientation of the optical axis based on the location of the center of the pupil in three-dimensional space relative to the center of curvature of the cornea.

[0023] Example 16:10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a location of a center of rotation of the eye based on a center of curvature of the cornea.

[0024] Example 17: 10. The display system of any preceding example, wherein the processing electronics is configured to determine a location of a center of rotation of the eye based on the location and orientation of the center of curvature of the cornea and the optical axis.

[0025] Example 18: 18. The display system of example 17, wherein the processing electronics is configured to determine the location of the center of rotation of the eye by translating a specific distance along the optical axis from the center of curvature of the cornea.

[0026] Example 19: The display system according to Example 18, wherein the specific distance from the center of curvature to the center of rotation is 4.0 mm to 6.0 mm.

[0027] Example 20: 20. The display system of claim 18 or 19, wherein the specific distance from the center of curvature to the center of rotation is about 4.7 mm.

[0028] Example 21: 20. The display system of example 18 or 19, wherein the specified distance is fixed.

[0029] Example 22: 20. The display system of claim 18 or 19, wherein the processing electronics is configured to determine a particular distance based at least on one or more images of the eye previously acquired using the one or more eye tracking cameras.

[0030] Example 23: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a location and orientation of the viewing axis offset from the optical axis based on the location and orientation of the optical axis.

[0031] Example 24: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine the location and orientation of the viewing axis based on an angular rotation about the optical axis.

[0032] Example 25: The display system of any of the preceding examples, wherein the processing electronics is configured to determine the location and orientation of the viewing axis based on an angular rotation of 4.0° to 6.5° relative to the optical axis.

[0033] Example 26: The display system of any of the preceding examples, wherein the processing electronics is configured to determine the location and orientation of the viewing axis based on an angular rotation of about 5.2° relative to the optical axis.

[0034] Example 27: 10. The display system of claim 1, wherein the processing electronics is configured to determine the location and orientation of the visual axis based on at least one image of the eye previously acquired using the one or more eye tracking cameras.

[0035] Example 28: 10. The display system of any preceding example, wherein the processing electronics is configured to determine a center of rotation of the eye based on multiple determinations of the location of the optical or visual axis over a period of time during which the eye is rotating.

[0036] Example 29: 10. The display system of any of the preceding examples, wherein the processing electronics is configured to determine the center of rotation by identifying areas of intersection, convergence, or proximity of multiple determinations of the location of the optical or visual axis over a period of time during which the eye is rotating.

[0037] Example 30:The display system of any of the preceding embodiments, wherein the processing electronics is configured to determine the user's vergence distances at which the user's left and right eyes are gazed upon based on determining the location and orientation of the optical axes relative to the user's left and right eyes.

[0038] Example 31: The display system of any of the preceding embodiments, wherein the processing electronics is configured to determine the user's convergence and divergence distances at which the user's left and right eyes are gazed based on determining the location and orientation of the visual axes relative to the user's left and right eyes.

[0039] Example 32: The display system of any of the preceding embodiments, wherein the processing electronics is configured to determine the convergence and divergence distances at which the user's left and right eyes are gazed based on identifying regions of intersection, convergence, or near-near of the visual axes for the user's left and right eyes.

[0040] Example 33: The display system of any of the preceding embodiments, wherein the processing electronics is configured to determine the convergence-divergence distance at which the user's left and right eyes are gazed by projecting the visual axes of the left and right eyes onto a horizontal plane and identifying areas of intersection, convergence, or near-nearby of the projections of the visual axes of the left and right eyes onto the horizontal plane.

[0041] Example 34: 10. The display system of any of the preceding embodiments, wherein the processing electronics is configured to determine a relative amount of at least one of divergence and collimation for projecting image content based on the determination of the vergence-divergence distance.

[0042] Example 35: 10. The display system of any preceding embodiment, wherein the processing electronics includes electronics on the frame.

[0043] Example 36:10. The display system of any preceding example, wherein the processing electronics includes electronics on the frame and electronics located remotely from the frame.

[0044] Example 37: 10. The display system of any preceding example embodiment, wherein the processing electronics includes electronics on the frame and electronics on a belt pack.

[0045] Example 38: A display system as described in any of the above examples, wherein at least a portion of the display is transparent and positioned in front of the user's eyes so that when the user wears the head-mounted display, the transparent portion transmits light from a portion of the environment in front of the user and the head-mounted display to the user's eyes and provides a view of a portion of the environment in front of the user and the head-mounted display.

[0046] Example 39: A display system as described in any of the above examples, wherein the head-mounted display receives light from a portion of the environment in front of the user at a first divergence amount and transmits light from a portion of the environment in front of the user's eyes to the user's eyes with a second divergence amount that is substantially similar to the first divergence amount.

[0047] Example 40: 10. The display system of any of the preceding examples, wherein the processing electronics is configured to obtain an estimate of the center of rotation by filtering, averaging, applying a Kalman filter to multiple estimated center of rotation positions, or any combination thereof.

[0048] Example 41: A display system as described in any of the above examples, wherein the processing electronics are configured to present the virtual image content to the user's eyes, rendered as if captured by a camera having an aperture at a determined position of the center of rotation of the user's eyes.

[0049] Example 42:10. The display system of any preceding example, wherein the processing electronics is configured to use a rendering camera at the center of rotation to render virtual images to be presented to the eyes.

[0050] Example 43: A display system as described in any of the above examples, wherein the processing electronics is configured to use a rendering camera configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye.

[0051] Example 44: A display system as described in any of the preceding examples, wherein the processing electronics is configured to use a rendering camera configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture at the center of rotation of the eye.

[0052] Example 45: A display system as described in any of the above examples, wherein the processing electronics is configured to render a virtual image to be presented to the eye using a rendering camera at the center of rotation, the rendering camera being modeled with an aperture at the center of rotation of the eye.

[0053] Example 46: 1. A display system configured to project light onto a user's eye and display virtual image content within the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea; a frame configured to be supported on a user's head; and a head-mounted display disposed on the frame, the display configured to project the light into the user's eyes with at least one of different amounts of divergence and collimation to display virtual image content in the user's field of view, such that the displayed virtual image content appears to arise from different depths and at different time periods; one or more eye-tracking cameras configured to image the eyes of a user; processing electronics in communication with the display and one or more eye-tracking cameras, the processing electronics configured to obtain a position estimate of a center of gaze of the eye based on images of the eye acquired using the one or more eye-tracking cameras, the center of gaze being estimated to be proximate to a pupil of the eye or between the cornea and the pupil of the eye; wherein the processing electronics is configured to present the virtual image content to the user's eyes, the virtual image content being rendered by a rendering camera located at the viewpoint center.

[0054] Example 47: The display system of any of the preceding examples, wherein the processing electronics is configured to present the virtual image content to the user's eyes, rendered as if captured by a camera having an aperture closer to the center of view than the retina.

[0055] Example 48: A display system as described in any of the above examples, wherein the processing electronics are configured to present the virtual image content to the user's eyes, rendered as if captured by a camera having an aperture closer to the center of viewpoint than the center of rotation of the eye.

[0056] Example 49: 10. The display system of claim 1, wherein the processing electronics is configured to present the virtual image content to the user's eyes, rendered as if captured by a camera having an aperture at the center of the viewpoint.

[0057] Example 50: 10. The display system of any of the preceding embodiments, wherein the center of gaze is not located at the pupil of the eye.

[0058] Example 51: 10. A display system as described in any of the preceding examples, wherein the processing electronics is configured to obtain estimates of the user's eye posture over time, and the processing electronics adjusts the position of the rendering camera based, at least in part, on the user's eye posture.

[0059] Example 52: A display system as described in any of the preceding examples, wherein the processing electronics are configured to track the user's eye posture over time, and the position of the rendering camera is adjusted over time in response to changes in the user's eye posture over time.

[0060] Example 53: 10. A display system as in any of the preceding examples, wherein the processing electronics is configured to obtain an estimate of the viewpoint center by filtering a plurality of estimated viewpoint center positions.

[0061] Example 54: 10. The display system of any of the preceding examples, wherein the processing electronics is configured to obtain an estimate of the gaze center by averaging multiple estimated gaze center positions and / or applying a Kalman filter.

[0062] Example 55: 10. The display system of any preceding example, wherein the center of gaze comprises a location within the anterior chamber of the user's eye.

[0063] Example 56: 10. The display system of any of the preceding examples, wherein a center of gaze comprises a position in front of the pupil of the user's eye.

[0064] Example 57: 10. The display system of any of the preceding embodiments, wherein the center of gaze comprises a position 1.0 mm to 2.0 mm in front of the pupil of the user's eye.

[0065] Example 58: 10. The display system of any of the preceding examples, wherein a center of gaze comprises a position approximately 1.0 mm in front of the pupil of the user's eye.

[0066] Example 59: 10. The display system of any of the preceding embodiments, wherein a center of gaze comprises a position 0.25 mm to 1.0 mm in front of the pupil of the user's eye.

[0067] Example 60: 10. The display system of any of the preceding embodiments, wherein a center of gaze comprises a position 0.5 mm to 1.0 mm in front of the pupil of the user's eye.

[0068] Example 61: 10. The display system of any of the previous examples, wherein a center of gaze comprises a position 0.25mm to 0.5mm in front of the pupil of the user's eye.

[0069] Example 62: 10. The display system of claim 1, wherein the center of gaze is along an optical axis of the eye, and the processing electronics is further configured to obtain a position estimate of the center of gaze by obtaining a position estimate of the optical axis of the eye.

[0070] Example 63: 10. The display system of claim 1, wherein the center of gaze is located along the optical axis of the eye, between the outer surface of the cornea of ​​the eye and the pupil, and the processing electronics is further configured to obtain a position estimate of the center of gaze by obtaining a position estimate of the optical axis of the eye.

[0071] Example 64:10. The display system of claim 9, wherein the center of gaze is located along the optical axis of the eye at a position between the outer surface of the cornea of ​​the eye and the pupil, and wherein the processing electronics is further configured to obtain a position estimate of the center of gaze by obtaining a position estimate of the optical axis of the eye and a position estimate of the center of rotation of the eye, the cornea of ​​the eye, the iris of the eye, the retina of the eye, and the pupil of the eye, or any combination thereof.

[0072] Example 65: 10. The display system of any preceding embodiment, wherein the processing electronics includes electronics on the frame.

[0073] Example 66: 10. The display system of any preceding example, wherein the processing electronics includes electronics on the frame and electronics located remotely from the frame.

[0074] Example 67: 10. The display system of any preceding example embodiment, wherein the processing electronics includes electronics on the frame and electronics on a belt pack.

[0075] Example 68: A display system as described in any of the above examples, wherein at least a portion of the display is transparent and positioned in front of the user's eyes so that when the user wears the head-mounted display, the transparent portion transmits light from a portion of the environment in front of the user and the head-mounted display to the user's eyes and provides a view of a portion of the environment in front of the user and the head-mounted display.

[0076] Example 69: A display system as described in any of the preceding examples, further comprising one or more light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more eye tracking cameras capture images of the eyes using light from the one or more light sources.

[0077] Example 70:10. The display system of any of the preceding examples, wherein the one or more light sources comprise at least two light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes.

[0078] Example 71: 10. The display system of any of the preceding examples, wherein the one or more light sources comprise at least three light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes.

[0079] Example 72: 10. The display system of any preceding embodiment, wherein the one or more light sources comprise infrared light emitters.

[0080] Example 73: 10. The display system of any of the preceding examples, wherein the one or more light sources form one or more flashes of light on the eye, and the processing electronics are configured to determine a position of a center of curvature of the cornea based on the one or more flashes of light.

[0081] Example 74: 10. The display system of any of the preceding examples, wherein the one or more light sources form one or more flashes of light on the eye, and the processing electronics are configured to determine a three-dimensional position of a center of curvature of the cornea based on the one or more flashes of light.

[0082] Example 75: 10. The display system of claim 1, wherein the one or more eye-tracking cameras are further configured to image a pupil of the user's eye, and the processing electronics are further configured to determine the position of the pupil of the eye based on at least an image of the pupil from the one or more eye-tracking cameras.

[0083] Example 76: 10. The display system of claim 1, wherein the one or more eye-tracking cameras are further configured to image a pupil of the user's eye, and the processing electronics are further configured to determine a three-dimensional position of the pupil of the eye based on at least an image of the pupil from the one or more eye-tracking cameras.

[0084] Example 77: The display system of any of the preceding examples, wherein the one or more eye-tracking cameras are further configured to image a pupil of the user's eye, and the processing electronics are further configured to determine the position of the pupil of the eye based on the position of the center of curvature of the cornea and based on images of the pupil from the one or more eye-tracking cameras.

[0085] Example 78: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine the optical axis of the eye based on a three-dimensional position of a center of curvature of the cornea and based on a three-dimensional position of the pupil.

[0086] Example 79: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a visual axis of the eye based on the optical axis.

[0087] Example 80: 10. The display system of any preceding example, wherein the processing electronics is configured to determine a visual axis of the eye based on an optical axis and a three-dimensional position of at least one of a center of curvature of the cornea, the pupil, or both.

[0088] Example 81: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a three-dimensional position of a center of rotation of the eye based on a three-dimensional position of a center of curvature of the cornea.

[0089] Example 82: 10. The display system of any preceding example, wherein the processing electronics is configured to determine a three-dimensional position of a center of rotation of the eye based on the three-dimensional position of the center of curvature of the cornea and based on the optical axis.

[0090] Example 83:10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a distance between the eye and the opposite eye of the user based on at least a three-dimensional position of a center of rotation of the eye.

[0091] Example 84: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine an interpupillary distance between the eye and the opposite eye of the user based on at least a three-dimensional position of a center of rotation of the eye.

[0092] Example 85: 10. The display system of any preceding embodiment, wherein the processing electronics is configured to determine the user's convergence distance based on at least the optical axes of the eyes.

[0093] Example 86: The display system of any of the preceding embodiments, wherein the processing electronics is configured to determine the user's convergence-divergence distance based on at least the determined optical axis of the user's eye and the optical axis of the opposite eye.

[0094] Example 87: 10. The display system of claim 1, wherein the processing electronics is configured to determine the user's convergence-divergence distance based on at least the determined visual axis of the user's eye and the opposite eye.

[0095] Example 88: 10. The display system of any preceding example, wherein the display is configured to project collimated light into the user's eye.

[0096] Example 89: 10. A display system according to any of the preceding examples, wherein the display is configured to project collimated light corresponding to image pixels into the user's eyes during a first period, and to project divergent light corresponding to the image pixels into the user's eyes during a second period.

[0097] Example 90: A display system as described in any of the above examples, wherein the display is configured to project light corresponding to image pixels having a first amount of divergence into the user's eye during a first period of time, and to project light corresponding to the image pixels having a second amount of divergence greater than the first amount of divergence into the user's eye during a second period of time.

[0098] Example 91: 1. A method of rendering virtual image content in a display system configured to project light onto a user's eye and display virtual image content within a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea; determining a position of a center of rotation of the eye using one or more eye tracking cameras configured to image the user's eye and track movement of the eye; Rendering virtual image content using a rendering engine with a rendering camera at a center of rotation of the eye, the rendering camera configured to render a virtual image to be presented to the eye; using a head-mounted display to project light into the user's eyes at different amounts of divergence so that virtual image content appears to arise from different depths and at different time periods, and displaying the rendered virtual image content in the user's field of view; A method comprising:

[0099] Example 92: 10. The method of any of the preceding examples, wherein the rendering camera is configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye.

[0100] Example 93:10. The method of any of the preceding examples, wherein the rendering camera is configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture at the center of rotation.

[0101] Example 94: 10. A method according to any of the preceding examples, wherein the rendering camera is modelled with an aperture at the centre of rotation of the eye.

[0102] Example 95: 10. The method of any of the preceding embodiments, wherein the rendering camera is modeled using an aperture, a lens, and a detector.

[0103] Example 96: 10. The method of claim 9, wherein the rendering camera has an aperture at a position along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of at least one of the iris or pupil.

[0104] Example 97: determining, with one or more eye-tracking cameras, a location of a center of gaze of the user's eye, the center of gaze of the user's eye being located less than about 1.0 mm from the pupil of the user's eye; Rendering the virtual image content using a rendering camera with a rendering engine; 4. The method of any of the preceding examples, further comprising: the rendering camera having an aperture at a determined position of a center of gaze of the user's eye.

[0105] Example 98: The method of any of the preceding examples, further comprising using a rendering engine to render the virtual image content using a rendering camera, the rendering camera having an aperture at a position along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of the center of gaze of the user's eye.

[0106] Example 99: determining, using processing electronics in communication with one or more eye-tracking cameras, a measure of change over time in the determined position of the center of gaze of the user's eyes; using the processing electronics, if it is determined that the measure of change over time exceeds a first threshold, instructing a rendering engine to render the virtual content using a rendering camera, the rendering camera having an aperture at the determined location of the center of rotation of the eye; 3. The method of any of the above examples, further comprising:

[0107] Example 100: The method of any of the preceding examples, further comprising, if it is determined using the processing electronics that the measured change over time is below a second threshold, instructing the rendering engine to render the virtual content using a rendering camera, the rendering camera having an aperture at the determined position of the eye center of gaze, and the first threshold indicating a higher level of change over time in the determined position of the user's eye center of gaze than a second threshold.

[0108] Example 101: 2. The method of claim 1, further comprising: if the processing electronics determine that the measured change over time is below a second threshold, instructing the rendering engine to render the virtual content using a rendering camera, the rendering camera having an aperture at the determined position of the eye's viewpoint center.

[0109] Example 102: 2. The method of claim 1, further comprising: if the processing electronics determine that the measured change over time is between a first threshold and a second threshold, instructing a rendering engine to render the virtual content using a rendering camera, the rendering camera having an aperture at a point along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of the center of gaze of the eye.

[0110] Example 103: The method according to any of the above examples, further comprising using at least a portion of the display, the portion being transparent and positioned in front of the user's eyes when the user is wearing the head-mounted display, to transmit light from a portion of the environment in front of the user and the head-mounted display to the user's eyes and provide a view of the portion of the environment in front of the user and the head-mounted display.

[0111] Example 104: 10. The method of any of the preceding examples, further comprising determining the position of at least one of the iris, pupil, or lens using one or more eye-tracking cameras.

[0112] Example 105: 10. The method of claim 1, further comprising using a rendering engine to render virtual image content using a rendering camera configured to present a virtual image to the eye, the virtual image rendered as if captured by a camera having an aperture located along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of at least one of the iris or pupil.

[0113] Example 106: determining, with one or more eye-tracking cameras, a location of a center of gaze of the user's eye, the center of gaze of the user's eye being located less than about 1.0 mm from the pupil of the user's eye; using a rendering engine to render virtual image content using a rendering camera configured to present a virtual image to the user's eye, the virtual image being rendered as if captured by a camera having an aperture at a determined position of a viewpoint center of the user's eye; 3. The method of any of the above examples, further comprising:

[0114] Example 107: The method of any of the above examples, further comprising using a rendering engine to render virtual image content using a rendering camera configured to present a virtual image to the eye, the virtual image rendered as if captured by a camera having an aperture at a position along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of the center of viewpoint of the user's eye.

[0115] Example 108: determining, using processing electronics in communication with one or more eye-tracking cameras, a measure of change over time in the determined position of the center of gaze of the user's eyes; if it is determined, using the processing electronics, that the measured change over time exceeds a first threshold, instructing a rendering engine to render the virtual content using a rendering camera as if it were captured by a camera having an aperture at the determined position of the center of rotation of the eye; 3. The method of any of the above examples, further comprising:

[0116] Example 109: The method of any of the above examples, further comprising instructing a rendering engine using the rendering camera to render the virtual content as if it were captured by a camera having an aperture at the determined position of the eye's center of gaze if the processing electronics determine that the measured change over time is below a second threshold, the first threshold indicating a higher level of change over time in the determined position of the user's eye's center of gaze than a second threshold.

[0117] Example 110: The method of any of the preceding examples, further comprising, if it is determined, using the processing electronics, that the measured change over time is below a second threshold, instructing a rendering engine to render the virtual content using a rendering camera as if it were captured by a camera having an aperture at the determined position of the center of gaze of the eye.

[0118] Example 111: The method of any of the above examples, further comprising, if the processing electronics determine that the measured change over time is between a first threshold and a second threshold, instructing a rendering engine to render the virtual content using a rendering camera as if it were captured by a camera having an aperture at a point along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of the center of gaze of the eye.

[0119] Example 112: 1. A display system configured to project light onto a user's eye and display virtual image content within the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea; a frame configured to be supported on a user's head; and a head-mounted display disposed on the frame, the display configured to project light at different amounts of divergence into the user's eyes and display virtual image content in the user's field of view, such that the displayed virtual image content appears to arise from different depths and for different time periods, the head-mounted display configured to project light having a first amount of divergence into the user's eyes in a first time period and to project light having a second amount of divergence into the user's eyes in a second time period, the first amount of divergence being different from the second amount of divergence; one or more eye-tracking cameras configured to image the eyes of a user; processing electronics in communication with the display and one or more eye tracking cameras, configured to: obtain an estimate of a center of rotation of the eye based on images of the eye obtained using the one or more eye tracking cameras; obtain an estimate of a convergence-divergence distance of the user based on images of the eye obtained using the one or more eye tracking cameras; and shift from projecting light into the user's eye at a first divergence amount to projecting light into the user's eye at a second divergence amount based on the estimated convergence-divergence distance of the user; A display system comprising:

[0120] Example 113: A display system as described in any of the above examples, wherein at least a portion of the display is transparent and positioned in front of the user's eyes so that when the user wears the head-mounted display, the transparent portion transmits light from a portion of the environment in front of the user and the head-mounted display to the user's eyes and provides a view of a portion of the environment in front of the user and the head-mounted display.

[0121] Example 114: 10. The display system of any preceding example embodiment, wherein the processing electronics is further configured to detect blinking of the eye based on images of the eye acquired using the one or more eye tracking cameras.

[0122] Example 115: 10. The display system of any preceding example embodiment, wherein the processing electronics is further configured to detect saccades of the eyes based on images of the eyes acquired using the one or more eye tracking cameras.

[0123] Example 116: The display system of any of the above embodiments, wherein the processing electronics is configured to shift from projecting light into the user's eye at a first divergence amount to projecting light into the user's eye at a second divergence amount based on the determined convergence-divergence distance of the user and based on whether the processing electronics detects an eye blink.

[0124] Example 117: A display system as described in any of the above examples, wherein the processing electronics is configured to shift from projecting light into the user's eye at a first divergence amount to projecting light into the user's eye at a second divergence amount based on the determined convergence-divergence movement distance of the user and based on whether the processing electronics detects a saccade of the eye.

[0125] Example 118: A display system as described in any of the above examples, wherein the processing electronics is configured to shift from projecting light into the user's eye at a first divergence amount to projecting light into the user's eye at a second divergence amount based on the determined convergence-divergence distance of the user and based on whether the processing electronics detects at least one of a saccade or a blink of the eye.

[0126] Example 119: A display system as described in any of the above embodiments, wherein the first divergence amount is associated with a convergence-divergence distance within a first range and the second divergence amount is associated with a convergence-divergence distance within a second range.

[0127] Example 120:10. A display system as described in any of the preceding embodiments, wherein the first divergence amount is associated with a convergence-divergence distance within a first range and the second divergence amount is associated with a convergence-divergence distance within a second range, the first and second ranges overlapping but not equal.

[0128] Example 121: The display system of any of the preceding embodiments, wherein the processing electronics is configured to shift from projecting light into the user's eyes at a first divergence amount to projecting light into the user's eyes at a second divergence amount in response to determining that the user's convergence-divergence distance is outside a first range and within a second range.

[0129] Example 122: The display system of any of the preceding embodiments, wherein the processing electronics is configured to shift from projecting light into the user's eyes at a first divergence amount to projecting light into the user's eyes at a second divergence amount in response to determining that the user's convergence-divergence distance is outside the second range and within the first range.

[0130] Example 123: The display system of any of the preceding embodiments, wherein the processing electronics is configured to shift from projecting light into the user's eye at a first divergence amount to projecting light into the user's eye at a second divergence amount in response to determining that the user's convergence-divergence distance is outside a first range and within a second range, and detecting an eye blink.

[0131] Example 124: The display system of any of the above embodiments, wherein the processing electronics is configured to shift from projecting light into the user's eye at a first divergence amount to projecting light into the user's eye at a second divergence amount in response to determining that the user's convergence-divergence distance is outside a first range and within a second range, and detecting a saccade of the eye.

[0132] Example 125:The display system of any of the preceding embodiments, wherein the processing electronics are configured to shift from projecting light into the user's eyes at a first divergence amount to projecting light into the user's eyes at a second divergence amount in response to determining that the user's convergence-divergence distance has been outside the first range and within the second range for a period of time longer than a predetermined period of time.

[0133] Example 126: The display system of any of the preceding embodiments, wherein the processing electronics are configured to shift from projecting light into the user's eyes at a first divergence amount to projecting light into the user's eyes at a second divergence amount in response to determining that the user's convergence-divergence distance is outside the first range and within the second range for a time period longer than a predetermined period of at least 10 seconds.

[0134] Example 127: 10. The display system of claim 1, wherein the head-mounted display comprises a first display element configured to project light having a first divergence amount and a second display element configured to project light having a second divergence amount.

[0135] Example 128: A display system as described in any of the above examples, wherein the display is configured to project light into the user's eye and display virtual image content in a discrete display mode, in which the display is configured to project light associated with multiple sequential frames using only one of the first display elements.

[0136] Example 129: A display system as described in any of the above examples, wherein the display is configured to project light into the user's eye and display virtual image content in a hybrid display mode, in which the display is configured to project light associated with multiple sequential frames using both the first and second display elements for each frame.

[0137] Example 130: The display system of any of the above examples, wherein the display is configured to project light into the user's eye and display virtual image content in a hybrid display mode, wherein the display is configured to project light associated with multiple sequential frames using both the first and second display elements for each frame, and wherein in the hybrid display mode the display is configured to project light using the first and second display elements, which is perceived by the user to have a given divergence amount between a first divergence amount and a second divergence amount.

[0138] Example 131: A display system as described in any of the above examples, wherein the display is configured to project light into the user's eyes and display virtual image content in a multifocal display mode, wherein the display is configured to project light associated with multiple sequential frames using both the first and second display elements for each frame, and in the multifocal display mode, the display is configured to project light associated with a first virtual image content at a third amount of divergence and project light associated with a second virtual image content at a fourth amount of divergence, the third amount of divergence being different from the fourth amount of divergence.

[0139] Example 132: 4. The display system of any of the preceding embodiments, wherein the third and fourth divergence amounts are each between the first and second divergence amounts.

[0140] Example 133: 10. The display system of any of the preceding embodiments, wherein at least one of the third and fourth divergence amounts is between the first and second divergence amounts.

[0141] Example 134: 4. The display system of any of the preceding embodiments, wherein the third and fourth divergence amounts are equal to the first and second divergence amounts, respectively.

[0142] Example 135: The display system of any of the preceding embodiments, wherein the display is configured to project light associated with a first virtual image into a first region of the user's field of view and project light associated with a second virtual image into a second region of the user's field of view, the first and second regions being different.

[0143] Example 136: A display system as described in any of the above examples, wherein the display is configured to project light associated with a first virtual image into a first region of the user's field of view and project light associated with a second virtual image into a second region of the user's field of view, the first and second regions not overlapping.

[0144] Example 137: a display system configured to project light to a user's left and right eyes and display virtual image content within the user's field of view, each eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea; a frame configured to be supported on a user's head; and a head-mounted display disposed on the frame, the display configured to project light into the user's left and right eyes with at least one of different amounts of divergence and collimation to display virtual image content in the user's field of view, such that the displayed virtual image content appears to originate from different distances and at different time periods from the user's left and right eyes; a first eye-tracking camera configured to image the left eye of the user; a second eye-tracking camera configured to image the user's right eye; and processing electronics in communication with the display and the first and second eye tracking cameras, the processing electronics configured to obtain an estimate of an interpupillary distance between the user's left and right eyes based on the left and right eye images acquired using the first and second eye tracking cameras; A display system comprising:

[0145] Example 138: A display system as described in any of the preceding examples, further comprising one or more light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes, wherein the one or more eye tracking cameras form images of the eyes using light from the one or more light sources.

[0146] Example 139: 10. The display system of any of the preceding examples, wherein the one or more light sources comprise at least two light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes.

[0147] Example 140: 10. The display system of any preceding embodiment, wherein the one or more light sources comprise infrared light emitters.

[0148] Example 141: 10. The display system of any preceding example, wherein the one or more light sources form one or more flashes of light on the eye, and the processing electronics are configured to determine the location of the cornea based on the one or more flashes of light.

[0149] Example 142: 10. The display system of any preceding example, wherein the cornea has a corneal sphere associated therewith, the cornea having a center of curvature, and the processing electronics is configured to determine a location of the center of curvature of the corneal sphere.

[0150] Example 143: 10. The display system of any of the preceding examples, wherein the cornea has a corneal sphere associated therewith, the cornea having a center of curvature, and the processing electronics are configured to determine a location of the center of curvature of the corneal sphere based on the one or more flashes of light.

[0151] Example 144: 10. The display system of any preceding example embodiment, wherein the one or more eye-tracking cameras are configured to image the pupils of the eyes.

[0152] Example 145: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a location of a center of the pupil.

[0153] Example 146: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine at least a portion of a boundary between the iris and the pupil.

[0154] Example 147: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a center of a boundary between the iris and the pupil.

[0155] Example 148: The display system of any of the preceding examples, wherein the processing electronics is configured to determine the location of the center of the pupil in three-dimensional space relative to the center of curvature of the cornea.

[0156] Example 149: The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a location and orientation of the optical axis.

[0157] Example 150: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine the location and orientation of the optical axis based on the location of the center of the pupil in three-dimensional space.

[0158] Example 151: 10. The display system of any preceding example, wherein the processing electronics is configured to determine the location and orientation of the optical axis based on the location of the center of the pupil in three-dimensional space relative to the center of curvature of the cornea.

[0159] Example 152: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a location of a center of rotation of the eye based on a center of curvature of the cornea.

[0160] Example 153: 10. The display system of any preceding example, wherein the processing electronics is configured to determine a location of a center of rotation of the eye based on the location and orientation of the center of curvature of the cornea and the optical axis.

[0161] Example 154: 10. The display system of any preceding example, wherein the processing electronics is configured to determine a location of the eye's center of rotation by translating the cornea a specific distance along the optical axis from its center of curvature.

[0162] Example 155: 1. A method of rendering virtual image content in a display system configured to project light to a user's left and right eyes and display virtual image content within the user's field of view, each eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea; determining a location of a center of rotation of the left eye and a location of a center of rotation of the right eye using one or more eye tracking cameras configured to image the user's eyes and track movement of the eyes; using processing electronics in communication with one or more eye-tracking cameras to estimate the user's interpupillary distance based on the determined positions of the centers of rotation of the left and right eyes; determining a current left eye pose and a current right eye pose using one or more eye tracking cameras; using processing electronics to estimate the user's current convergence-divergence distance by comparing the estimated interpupillary distance with the determined current left eye posture and the determined current right eye posture; A method comprising:

[0163] Example 156: 4. The method according to any of the preceding examples, wherein determining the current left and right eye pose comprises estimating the position of the pupil of the user's left eye and the position of the pupil of the user's right eye using one or more eye tracking cameras.

[0164] Example 157: 4. The method according to any of the preceding examples, wherein determining the current left and right eye pose comprises estimating the position of the cornea of ​​the user's left eye and the position of the cornea of ​​the user's right eye using one or more eye tracking cameras.

[0165] Example 158: 4. The method according to any of the preceding examples, wherein determining the current left and right eye pose comprises estimating the position of the iris of the user's left eye and the position of the iris of the user's right eye using one or more eye tracking cameras.

[0166] Example 159: 4. The method according to any of the preceding examples, wherein determining the current left and right eye pose comprises estimating the position of the lens of the user's left eye and the position of the lens of the user's right eye using one or more eye tracking cameras.

[0167] Example 160: The step of estimating a current convergence-divergence movement distance of the user includes: estimating, using processing electronics, the distance between the positions of the irises of the user's left and right eyes; using processing electronics to estimate a current convergence-divergence distance of the user based on a comparison of the estimated interpupillary distance and an estimated distance between the positions of the irises of the user's left and right eyes; 2. The method of any of the above examples, comprising:

[0168] Example 161: The method of any of the above examples, further comprising using a head-mounted display to project light into the user's eyes at different amounts of divergence so that virtual image content appears to arise from different depths and at different time periods, and displaying the rendered virtual image content in the user's field of view.

[0169] Example 162:The method according to any of the above examples, further comprising using at least a portion of the display, the portion being transparent and positioned in front of the user's eyes when the user is wearing the head-mounted display, to transmit light from a portion of the environment in front of the user and the head-mounted display to the user's eyes and provide a view of the portion of the environment in front of the user and the head-mounted display.

[0170] Example 163: 1. A display system configured to project light onto a user's eye and display virtual image content within the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea; a frame configured to be supported on a user's head; and a head-mounted display disposed on the frame, the display configured to project the light into the user's eyes with at least one of different amounts of divergence and collimation to display virtual image content in the user's field of view, such that the displayed virtual image content appears to arise from different depths and at different time periods; one or more eye-tracking cameras configured to image the eyes of a user; processing electronics in communication with the display and one or more eye tracking cameras, the processing electronics configured to obtain a position estimate of a center of rotation of the eye based on images of the eye acquired using the one or more eye tracking cameras, and to obtain a direction estimate of an optical axis of the eye based on the images; wherein the processing electronics are configured to present the virtual image content to the user's eye, rendered as if captured by a camera having an aperture positioned along an optical axis and spaced from an estimated location of the eye's center of rotation such that the camera is between 6.0 mm and 13.0 mm away from the retina in a direction.

[0171] Example 164:The display system of any of the above examples, wherein the processing electronics is configured to present the virtual image content to the user's eye, rendered as if captured by a camera having an aperture positioned along an optical axis and spaced 7.0 mm to 12.0 mm in a direction from the retina and spaced from the estimated location of the eye's center of rotation.

[0172] Example 165: The display system of any of the above examples, wherein the processing electronics is configured to present the virtual image content to the user's eye, rendered as if captured by a camera having an aperture positioned along an optical axis and spaced 8.0 mm to 11.0 mm in a direction from the retina and spaced from the estimated location of the eye's center of rotation.

[0173] Example 166: The display system of any of the above examples, wherein the processing electronics is configured to present the virtual image content to the user's eye, rendered as if captured by a camera having an aperture positioned along an optical axis and spaced 9.0 mm to 10.0 mm in a direction from the retina and spaced from the estimated location of the eye's center of rotation.

[0174] Example 167: The display system of any of the preceding examples, wherein the processing electronics is configured to present the virtual image content to the user's eye, rendered as if captured by a camera having an aperture positioned along an optical axis and spaced 9.5 mm to 10.0 mm in a direction from the retina and spaced from the estimated location of the eye's center of rotation.

[0175] Example 168: A display system as described in any of the above examples, wherein the processing electronics is configured to present the virtual image content to the user's eye, rendered as if captured by a camera having an aperture positioned along the optical axis and spaced approximately 9.7 mm from the estimated location of the eye's center of rotation.

[0176] Example 169: 10. The display system of any preceding embodiment, wherein the processing electronics includes electronics on the frame.

[0177] Example 170: 10. The display system of any preceding example, wherein the processing electronics includes electronics on the frame and electronics located remotely from the frame.

[0178] Example 171: 10. The display system of any preceding example embodiment, wherein the processing electronics includes electronics on the frame and electronics on a belt pack.

[0179] Example 172: A display system as described in any of the above examples, wherein at least a portion of the display is transparent and positioned in front of the user's eyes so that when the user wears the head-mounted display, the transparent portion transmits light from a portion of the environment in front of the user and the head-mounted display to the user's eyes and provides a view of a portion of the environment in front of the user and the head-mounted display.

[0180] Example 173: A display system as described in any of the preceding examples, further comprising one or more light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more eye tracking cameras capture images of the eyes using light from the one or more light sources.

[0181] Example 174: 10. The display system of any of the preceding examples, wherein the one or more light sources comprise at least two light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes.

[0182] Example 175: 10. The display system of any of the preceding examples, wherein the one or more light sources comprise at least three light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes.

[0183] Example 176: 10. The display system of any preceding embodiment, wherein the one or more light sources comprise infrared light emitters.

[0184] Example 177: 1. A display system configured to project light onto a user's eye and display virtual image content within the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea; a frame configured to be supported on a user's head; and a head-mounted display disposed on the frame, the display configured to project the light into the user's eyes with at least one of different amounts of divergence and collimation to display virtual image content in the user's field of view, such that the displayed virtual image content appears to arise from different depths and at different time periods; one or more eye-tracking cameras configured to image the eyes of a user; processing electronics in communication with the display and the one or more eye tracking cameras; wherein the processing electronics is configured to present the virtual image content to the user's eye, the virtual image content being rendered by a rendering camera located at the pupil of the eye or between the pupil and the cornea of ​​the eye.

[0185] Example 178: The display system according to any of the previous embodiments, wherein the rendering camera is positioned 1.0 mm to 2.0 mm in front of the pupil of the user's eye.

[0186] Example 179: 10. The display system of any of the preceding embodiments, wherein the rendering camera is positioned approximately 1.0 mm in front of the pupil of the user's eye.

[0187] Example 180:10. The display system of any of the preceding embodiments, wherein the rendering camera is positioned 0.25 mm to 1.0 mm in front of the pupil of the user's eye.

[0188] Example 181: 10. The display system of any of the preceding embodiments, wherein the rendering camera is positioned 0.5 mm to 1.0 mm in front of the pupil of the user's eye.

[0189] Example 182: 10. The display system of any of the previous embodiments, wherein the rendering camera is positioned 0.25 mm to 0.5 mm in front of the pupil of the user's eye.

[0190] Example 183: 10. The display system of any of the previous examples, wherein the rendering camera is located at the pupil of the eye.

[0191] Example 184: 10. The display system of any of the previous examples, wherein the rendering camera is not located at the pupil of the eye.

[0192] Example 185: 10. The display system of any of the previous examples, wherein the camera comprises a pinhole camera.

[0193] Example 186: 10. The display system of any of the previous examples, wherein the aperture comprises a pinhole of a pinhole camera.

[0194] Example 187: 1. A method of rendering virtual image content in a display system configured to project light onto a user's eye and display virtual image content within a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea; determining, using one or more eye cameras configured to image the user's eyes, a position based on imaging of the eyes using the one or more cameras; and rendering, using a rendering engine, virtual image content using a rendering camera at a location based on the determined position, the rendering camera configured to render a virtual image to be presented to the eyes; using a head mounted display to project light into the user's eye and display rendered virtual image content in the user's field of view; A method comprising:

[0195] Example 188: 10. The method according to any of the preceding examples, wherein the location is the center of rotation of the eye.

[0196] Example 189: 10. The method according to any of the preceding embodiments, wherein the location of the rendering camera is at the center of rotation of the eye.

[0197] Example 190: 10. The method according to any of the preceding examples, wherein the location is the center of gaze of the eye.

[0198] Example 191: 10. The method according to any of the preceding embodiments, wherein the rendering camera is located at the eye center.

[0199] Example 192: 10. The method of any of the preceding examples, wherein the rendering camera is configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye.

[0200] Example 193: 10. The method of any of the preceding examples, wherein the rendering camera is configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture at the center of rotation.

[0201] Example 194: 10. A method according to any of the preceding examples, wherein the rendering camera is modelled with an aperture at the centre of rotation of the eye.

[0202] Example 195: 10. The method of any of the preceding embodiments, wherein the rendering camera is modeled using an aperture, a lens, and a detector.

[0203] Example 196: 10. The method of claim 9, wherein the rendering camera has an aperture at a position along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of at least one of the iris or pupil.

[0204] Example 197: determining, using one or more cameras, a location of a center of view of the user's eye, the center of view of the user's eye being located less than about 1.0 mm from the pupil of the user's eye; Rendering the virtual image content using a rendering camera with a rendering engine; 10. The method of any of the preceding embodiments, further comprising: a rendering camera having an aperture at the determined position of the user's eye center of view.

[0205] Example 198: The method of any of the preceding examples, further comprising using a rendering engine to render the virtual image content using a rendering camera, the rendering camera having an aperture at a position along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of the center of gaze of the user's eye.

[0206] Example 199: determining, using processing electronics in communication with the one or more cameras, a measure of change over time in the determined position of the center of gaze of the user's eyes; using the processing electronics, if it is determined that the measure of change over time exceeds a first threshold, instructing a rendering engine to render the virtual content using a rendering camera, the rendering camera having an aperture at the determined location of the center of rotation of the eye; 3. The method of any of the above examples, further comprising:

[0207] Example 200: The method of any of the preceding examples, further comprising, if it is determined using the processing electronics that the measured change over time is below a second threshold, instructing the rendering engine to render the virtual content using a rendering camera, the rendering camera having an aperture at the determined position of the eye center of gaze, and the first threshold indicating a higher level of change over time in the determined position of the user's eye center of gaze than a second threshold.

[0208] Example 201: 2. The method of claim 1, further comprising: if the processing electronics determine that the measured change over time is below a second threshold, instructing the rendering engine to render the virtual content using a rendering camera, the rendering camera having an aperture at the determined position of the eye's viewpoint center.

[0209] Example 202: 2. The method of claim 1, further comprising: if the processing electronics determine that the measured change over time is between a first threshold and a second threshold, instructing a rendering engine to render the virtual content using a rendering camera, the rendering camera having an aperture at a point along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of the center of gaze of the eye.

[0210] Example 203: The method according to any of the above examples, further comprising using at least a portion of the display, the portion being transparent and positioned in front of the user's eyes when the user is wearing the head-mounted display, to transmit light from a portion of the environment in front of the user and the head-mounted display to the user's eyes and provide a view of the portion of the environment in front of the user and the head-mounted display.

[0211] Example 204: The method of any of the preceding examples, further comprising determining the position of at least one of the iris, pupil, or lens using one or more cameras.

[0212] Example 205: 10. The method of claim 1, further comprising using a rendering engine to render virtual image content using a rendering camera configured to present a virtual image to the eye, the virtual image rendered as if captured by a camera having an aperture located along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of at least one of the iris or pupil.

[0213] Example 206: determining, using one or more cameras, a location of a center of view of the user's eye, the center of view of the user's eye being located less than about 1.0 mm from the pupil of the user's eye; using a rendering engine to render virtual image content using a rendering camera configured to present a virtual image to the user's eye, the virtual image being rendered as if captured by a camera having an aperture at a determined position of a viewpoint center of the user's eye; 3. The method of any of the above examples, further comprising:

[0214] Example 207: The method of any of the above examples, further comprising using a rendering engine to render virtual image content using a rendering camera configured to present a virtual image to the eye, the virtual image rendered as if captured by a camera having an aperture at a position along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of the center of viewpoint of the user's eye.

[0215] Example 208: A method according to any of the above examples, wherein using the head-mounted display to project light into the user's eyes and display rendered virtual image content in the user's field of view includes projecting light into the user's eyes and displaying rendered virtual image content in the user's field of view at different amounts of divergence so that the virtual image content appears to arise from different depths and at different time periods.

[0216] Example 209: 10. The method of any of the preceding embodiments, wherein the different amounts of divergence include zero divergence.

[0217] Example 210: 10. The method of any of the preceding embodiments, wherein the different amounts of divergence include collimation.

[0218] Example 211: 1. A display system configured to project light onto a user's eye and display virtual image content within the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea; a frame configured to be supported on a user's head; and a head-mounted display disposed on the frame, the display configured to project light into the user's eye and display virtual image content in the user's field of view; one or more cameras configured to image the user's eyes; processing electronics in communication with the display and the one or more cameras, the processing electronics configured to obtain a position of the eye based on images of the eye obtained using the one or more cameras; wherein the processing electronics is configured to present the virtual image content to the user's eyes, the virtual image content being rendered by a rendering camera located at a location based on the determined position.

[0219] Example 212: 10. The display system of any of the preceding embodiments, wherein the position is an estimate of the center of rotation of the eye.

[0220] Example 213: 10. The display system of any of the previous examples, wherein the rendering camera is located at the estimated center of rotation of the eye.

[0221] Example 214: 10. A display system according to any of the preceding embodiments, wherein the position is an estimate of the center of gaze of the eye.

[0222] Example 215: 10. A display system according to any of the preceding embodiments, wherein the location of the rendering camera is at the estimated center of view of the eye.

[0223] Example 216: The display system of any of the preceding examples, wherein the processing electronics is configured to present the virtual image content to the user's eye, rendered as if captured by a camera having an aperture closer to the center of view than the retina of the eye.

[0224] Example 217: The display system of any of the preceding examples, wherein the processing electronics is configured to present the virtual image content to the user's eyes, rendered as if captured by a camera having an aperture closer to the center of rotation than the retina.

[0225] Example 218:10. The display system of claim 1, wherein the processing electronics is configured to present the virtual image content to the user's eyes, rendered as if captured by a camera having an aperture at the center of rotation.

[0226] Example 219: The display system of any of the preceding examples, wherein the processing electronics is configured to present the virtual image content to the user's eyes, rendered as if captured by a camera having an aperture closer to the center of rotation than the center of viewpoint.

[0227] Example 220: The display system of any of the preceding examples, wherein the processing electronics is configured to present the virtual image content to the user's eyes, rendered as if captured by a camera having an aperture closer to the center of viewpoint than the retina.

[0228] Example 221: A display system as described in any of the above examples, wherein the processing electronics are configured to present the virtual image content to the user's eyes, rendered as if captured by a camera having an aperture closer to the center of viewpoint than the center of rotation of the eye.

[0229] Example 222: 10. The display system of claim 1, wherein the processing electronics is configured to present the virtual image content to the user's eyes, rendered as if captured by a camera having an aperture at the center of the viewpoint.

[0230] Example 223: 10. The display system of any of the preceding embodiments, wherein the center of gaze is not located at the pupil of the eye.

[0231] Example 224:10. A display system as described in any of the preceding examples, wherein the processing electronics is configured to obtain estimates of the user's eye posture over time, and the processing electronics adjusts the position of the rendering camera based, at least in part, on the user's eye posture.

[0232] Example 225: A display system as described in any of the above examples, wherein the processing electronics are configured to track the user's eye posture over time, and the position of the rendering camera is adjusted over time in response to changes in the user's eye posture over time.

[0233] Example 226: 10. A display system as in any of the preceding examples, wherein the processing electronics is configured to obtain an estimate of the viewpoint center by filtering a plurality of estimated viewpoint center positions.

[0234] Example 227: 10. The display system of any of the preceding examples, wherein the processing electronics is configured to obtain an estimate of the gaze center by averaging multiple estimated gaze center positions and / or applying a Kalman filter.

[0235] Example 228: 10. The display system of any preceding example, wherein the center of gaze comprises a location within the anterior chamber of the user's eye.

[0236] Example 229: 10. The display system of any of the preceding examples, wherein a center of gaze comprises a position in front of the pupil of the user's eye.

[0237] Example 230: 10. The display system of any of the preceding embodiments, wherein the center of gaze comprises a position 1.0 mm to 2.0 mm in front of the pupil of the user's eye.

[0238] Example 231: 10. The display system of any of the preceding examples, wherein a center of gaze comprises a position approximately 1.0 mm in front of the pupil of the user's eye.

[0239] Example 232: 10. The display system of any of the preceding embodiments, wherein a center of gaze comprises a position 0.25 mm to 1.0 mm in front of the pupil of the user's eye.

[0240] Example 233: 10. The display system of any of the preceding embodiments, wherein a center of gaze comprises a position 0.5 mm to 1.0 mm in front of the pupil of the user's eye.

[0241] Example 234: 10. The display system of any of the previous examples, wherein a center of gaze comprises a position 0.25mm to 0.5mm in front of the pupil of the user's eye.

[0242] Example 235: 10. The display system of claim 1, wherein the center of gaze is along an optical axis of the eye, and the processing electronics is further configured to obtain a position estimate of the center of gaze by obtaining a position estimate of the optical axis of the eye.

[0243] Example 236: 10. The display system of claim 1, wherein the center of gaze is located along the optical axis of the eye, between the outer surface of the cornea of ​​the eye and the pupil, and the processing electronics is further configured to obtain a position estimate of the center of gaze by obtaining a position estimate of the optical axis of the eye.

[0244] Example 237: 10. The display system of claim 9, wherein the center of gaze is located along the optical axis of the eye at a position between the outer surface of the cornea of ​​the eye and the pupil, and wherein the processing electronics is further configured to obtain a position estimate of the center of gaze by obtaining a position estimate of the optical axis of the eye and a position estimate of the center of rotation of the eye, the cornea of ​​the eye, the iris of the eye, the retina of the eye, and the pupil of the eye, or any combination thereof.

[0245] Example 238:10. The display system of any preceding embodiment, wherein the processing electronics includes electronics on the frame.

[0246] Example 239: 10. The display system of any preceding example, wherein the processing electronics includes electronics on the frame and electronics located remotely from the frame.

[0247] Example 240: 10. The display system of any preceding example embodiment, wherein the processing electronics includes electronics on the frame and electronics on a belt pack.

[0248] Example 241: A display system as described in any of the above examples, wherein at least a portion of the display is transparent and positioned in front of the user's eyes so that when the user wears the head-mounted display, the transparent portion transmits light from a portion of the environment in front of the user and the head-mounted display to the user's eyes and provides a view of a portion of the environment in front of the user and the head-mounted display.

[0249] Example 242: A display system as described in any of the above examples, further comprising one or more light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more cameras capturing images of the eyes using light from the one or more light sources.

[0250] Example 243: 10. The display system of any of the preceding examples, wherein the one or more light sources comprise at least two light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes.

[0251] Example 244: 10. The display system of any of the preceding examples, wherein the one or more light sources comprise at least three light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes.

[0252] Example 245: 10. The display system of any preceding embodiment, wherein the one or more light sources comprise infrared light emitters.

[0253] Example 246: 10. The display system of any of the preceding examples, wherein the one or more light sources form one or more flashes of light on the eye, and the processing electronics are configured to determine a position of a center of curvature of the cornea based on the one or more flashes of light.

[0254] Example 247: 10. The display system of any of the preceding examples, wherein the one or more light sources form one or more flashes of light on the eye, and the processing electronics are configured to determine a three-dimensional position of a center of curvature of the cornea based on the one or more flashes of light.

[0255] Example 248: 10. The display system of claim 1, wherein the one or more cameras are further configured to image a pupil of the user's eye, and the processing electronics are further configured to determine a position of the pupil of the eye based on at least an image of the pupil from the one or more cameras.

[0256] Example 249: 10. The display system of claim 1, wherein the one or more cameras are further configured to image a pupil of the user's eye, and the processing electronics are further configured to determine a three-dimensional position of the pupil of the eye based on at least an image of the pupil from the one or more cameras.

[0257] Example 250: 10. The display system of claim 1, wherein the one or more cameras are further configured to image a pupil of the user's eye, and the processing electronics are further configured to determine the position of the pupil of the eye based on the position of the center of curvature of the cornea and based on images of the pupil from the one or more cameras.

[0258] Example 251:10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine the optical axis of the eye based on a three-dimensional position of a center of curvature of the cornea and based on a three-dimensional position of the pupil.

[0259] Example 252: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a visual axis of the eye based on the optical axis.

[0260] Example 253: 10. The display system of any preceding example, wherein the processing electronics is configured to determine a visual axis of the eye based on an optical axis and a three-dimensional position of at least one of a center of curvature of the cornea, the pupil, or both.

[0261] Example 254: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a three-dimensional position of a center of rotation of the eye based on a three-dimensional position of a center of curvature of the cornea.

[0262] Example 255: 10. The display system of any preceding example, wherein the processing electronics is configured to determine a three-dimensional position of a center of rotation of the eye based on the three-dimensional position of the center of curvature of the cornea and based on the optical axis.

[0263] Example 256: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine a distance between the eye and the opposite eye of the user based on at least a three-dimensional position of a center of rotation of the eye.

[0264] Example 257: 10. The display system of any preceding example embodiment, wherein the processing electronics is configured to determine an interpupillary distance between the eye and the opposite eye of the user based on at least a three-dimensional position of a center of rotation of the eye.

[0265] Example 258:10. The display system of any preceding embodiment, wherein the processing electronics is configured to determine the user's convergence distance based on at least the optical axes of the eyes.

[0266] Example 259: The display system of any of the preceding embodiments, wherein the processing electronics is configured to determine the user's convergence-divergence distance based on at least the determined optical axis of the user's eye and the optical axis of the opposite eye.

[0267] Example 260: 10. The display system of claim 1, wherein the processing electronics is configured to determine the user's convergence-divergence distance based on at least the determined visual axis of the user's eye and the opposite eye.

[0268] Example 261: 10. The display system of any preceding example, wherein the display is configured to project collimated light into the user's eye.

[0269] Example 262: 10. A display system according to any of the preceding examples, wherein the display is configured to project collimated light corresponding to image pixels into the user's eyes during a first period, and to project divergent light corresponding to the image pixels into the user's eyes during a second period.

[0270] Example 263: A display system as described in any of the above examples, wherein the display is configured to project light corresponding to image pixels having a first amount of divergence into the user's eye during a first period of time, and to project light corresponding to the image pixels having a second amount of divergence greater than the first amount of divergence into the user's eye during a second period of time.

[0271] Example 264: 10. A display system according to any of the preceding examples, wherein the centre of gaze is estimated to be proximate the pupil of the eye.

[0272] Example 265: 10. A display system according to any of the preceding examples, wherein a centre of gaze is estimated to be between the cornea and the pupil of the eye.

[0273] Example 266: A display system as described in any of the above examples, wherein the display is configured to project light into the user's eye with at least one of different amounts of divergence and collimation and display virtual image content in the user's field of view, such that the displayed virtual image content appears to arise from different depths and at different time periods.

[0274] Example 267: 1. A display system configured to project light onto a user's eye and display virtual image content within the user's field of view, the eye having a cornea, an iris, a pupil, a lens, and a retina; a frame configured to be supported on a user's head; and a head-mounted display disposed on the frame, the display configured to project the light into the user's eyes with at least one of different amounts of divergence and collimation to display virtual image content in the user's field of view, such that the displayed virtual image content appears to arise from different depths and at different time periods; one or more eye-tracking cameras configured to image the eyes of a user; processing electronics in communication with the display and the one or more eye-tracking cameras, obtaining a position and orientation estimate of an optical axis extending through the lens, pupil, and cornea of ​​the user's eye based on images of the eye obtained using one or more eye-tracking cameras; identifying a particular location along an axis in rendering space that is aligned with the optical axis of the eye; A virtual rendering camera located at a specific point in the rendering space presenting the virtual image content to the user's eye, as rendered by processing electronics configured to A display system comprising:

[0275] Example 268: The display system of Example 267, wherein the head-mounted display is configured to project light having a first amount of divergence into the user's eye during a first period of time and project light having a second amount of divergence into the user's eye during a second period of time, the first amount of divergence being different from the second amount of divergence, and the processing electronics is further configured to obtain an estimate of the user's convergence-divergence distance based on images of the eye acquired using the one or more eye tracking cameras, and shift the projection of light into the user's eye from the first amount of divergence to the second amount of divergence based on the estimated convergence-divergence distance of the user.

[0276] Example 269: The display system of Example 267, wherein the head-mounted display is configured to project light having a first amount of divergence into the user's eye during a first period of time and project light having a second amount of divergence into the user's eye during a second period of time, the first amount of divergence being different from the second amount of divergence, and the processing electronics is further configured to obtain an estimate of the user's convergence-divergence distance based on images of the eye acquired using the one or more eye tracking cameras, and shift the projection of light into the user's eye from the first amount of divergence to the second amount of divergence based on the estimated convergence-divergence distance of the user.

[0277] Example 270: the processing electronics are further configured to determine, based on images of the eye acquired using one or more eye-tracking cameras, a location along the optical axis at which a center of rotation of the eye is estimated to be located; the processing electronics are configured to identify a particular location along an axis in a rendering space that is aligned with the optical axis of the eye based on a location along the optical axis at which the center of rotation of the eye is estimated to be located. The display system of Example 267.

[0278] Example 271: A display system as described in Example 267, wherein a particular location along an axis in the rendering space comprises a location along an axis in the rendering space at which it is determined that the parallax shift in the rendering space is reduced.

[0279] Example 272: A display system as described in Example 267, wherein a particular location along an axis in the rendering space comprises a location along an axis in the rendering space that is determined to minimize parallax shift in the rendering space.

[0280] Example 273: The display system of any of the preceding examples, wherein the processing electronics is configured to obtain an estimate of the center of rotation of the eye based on determining multiple gaze directions of the user's eye over a period of time as the eye rotates, based on images of the eye acquired by the one or more eye tracking cameras.

[0281] Example 274: The display system of Example 273, wherein the processing electronics is configured to determine the gaze direction based on the amount of variation in the shape of one or more of the pupil, iris, or limbus of the user's eye in images acquired using the one or more eye tracking cameras over a period of time during which the eye is rotating.

[0282] Example 275: 10. The display system of claim 1, wherein the processing electronics is configured to determine an array of positions based on a plurality of spatial locations on an image of the user's eyes acquired using the one or more eye tracking cameras.

[0283] Example 276: 276. The display system of Example 275, wherein the array of positions corresponds to at least a portion of an ellipse.

[0284] Example 277:A display system as described in either of Examples 275 or 276, wherein the processing electronics is configured to determine the array of positions by fitting a curve to multiple spatial locations on the image of the user's eye.

[0285] Example 278: 278. The display system of Example 277, wherein the curve comprises an ellipse.

[0286] Example 279: A display system described in any of Examples 275-278, wherein the multiple spatial locations on the image include spatial locations on the limbus of the user's eye within the image.

[0287] Example 280: A display system described in any of Examples 275-279, wherein the multiple spatial locations on the image include spatial locations on the boundary between the iris and sclera of the user's eye within the image.

[0288] Example 281: A display system described in any of Examples 275-279, wherein the multiple spatial locations on the image include spatial locations on the boundary between the cornea and sclera of the user's eye in the image acquired using the one or more eye tracking cameras.

[0289] Example 282: A display system described in any of Examples 275-281, wherein the processing electronics is configured to determine a plurality of linear paths extending from locations on a first side of the array of positions, through the array of positions, to a second opposite side of the array of positions.

[0290] Example 283: The display system of Example 282, wherein the processing electronics is configured to determine a circular region based on the plurality of linear paths, the circular region having a radius R.

[0291] Example 284: The display system of Example 283, wherein the radius R corresponds to the average radius of the corneal limbus.

[0292] Example 285: A display system as described in Example 283, wherein the radius R corresponds to the measured radius of the limbus of the user's eye.

[0293] Example 286: 284. The display system of claim 283, wherein the radius R corresponds to the average radius of the pupil.

[0294] Example 287: A display system as described in Example 283, wherein the radius R corresponds to the measured radius of the pupil of the user's eye.

[0295] Example 288: A display system described in any of Examples 282-287, wherein the processing electronics are configured to determine the location and direction of a normal passing through a central portion of the circular region.

[0296] Example 289: A display system described in any of Examples 282-288, wherein the processing electronics is configured to determine individual locations and directions of multiple normals passing through central portions of individual circular regions based on multiple images of the eye previously acquired using the one or more eye tracking cameras.

[0297] Example 290: The display system of Example 289, wherein the processing electronics is configured to determine a position where the multiple normals converge or intersect.

[0298] Example 291: The display system of Example 289, wherein the processing electronics are configured to obtain an estimate of the center of rotation of the user's eye by identifying areas of intersection, convergence, or close proximity of multiple of the normals, determined based on images of the user's eye acquired over a period in which the eye is rotating.

[0299] Example 292:A display system as described in Example 289, wherein the processing electronics are configured to obtain an estimate of the center of rotation of the user's eye based on the locations and orientations of multiple ones of the multiple normals determined based on images of the user's eye acquired over a period of time while the eye is rotating.

[0300] Example 293: A display system described in any of Examples 282-292, wherein the location on the first side of the array position corresponds to the origin of the coordinate system of one of the one or more eye tracking cameras.

[0301] Example 294: 10. The display system of any of the preceding examples, wherein the processing electronics is configured to obtain an estimate of the center of rotation by filtering, averaging, applying a Kalman filter to multiple estimated center of rotation positions, or any combination thereof.

[0302] Various additional embodiments of display systems are described herein, such as the additional embodiments listed below, that project light onto one or more users' eyes and display virtual image content within the user's field of view.

[0303] Additional Example 1: 1. A display system configured to project light onto a user's eye and display virtual image content within the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea; a frame configured to be supported on a user's head; and a head-mounted display disposed on the frame, the display configured to project light into the user's eyes and display virtual image content in the user's field of view, at least a portion of the display being transparent and positioned in a location in front of the user's eyes such that, when the user wears the frame, the transparent portion transmits light from a portion of the environment in front of the user and the head-mounted display to the user's eyes and provides a view of the portion of the environment in front of the user and the head-mounted display; an inward-facing imaging system configured to image an eye of a user; processing electronics in communication with the inward-facing imaging system, the processing electronics configured to obtain an estimate of a center of rotation of the eye based on a plurality of images of the eye acquired with the inward-facing imaging system, the processing electronics configured to determine an amount of variability in calculated values ​​of the center of rotation, and select a statistically determined estimate of the center of rotation based on the amount of variability; A display system comprising:

[0304] Additional Example 2: 10. The display system of Additional Example 1, wherein a reduced amount of variation is used to identify the statistically determined center of rotation.

[0305] Additional Example 3: 3. The display system of claim 1, wherein a first set of estimates of the center of rotation are calculated based on first values ​​of parameters used to calculate the center of rotation, and a first amount of variation is determined from the first set of estimates.

[0306] Additional Example 4: A display system as described in additional example 3, wherein a second set of estimates of the center of rotation are calculated based on second values ​​of the parameters, and a second amount of variation is determined from the second set of estimates.

[0307] Additional Example 5:The display system of additional example 4, wherein the first and second variations are compared to determine a set having a reduced variation, and the determination of the statistically determined estimate of the center of rotation is based on this comparison.

[0308] Additional Example 6: 3. The display system of any one of additional examples 1 to 2, wherein the plurality of sets of values ​​for the center of rotation are calculated based on a plurality of distinct values ​​for the parameter and distinct amounts of variation determined for the different distinct sets.

[0309] Additional Example 7: 7. A display system as described in additional example 6, wherein the individual variations are compared to determine a set having a reduced variation, and the determination of the statistically determined estimate of the center of rotation is based on the comparison.

[0310] Additional Example 8: The display system of additional example 6 or 7, wherein the parameter values ​​for the set having the lowest amount of variation are used in calculating the statistically determined estimate of the center of rotation.

[0311] Additional Example 9: The display system of additional example embodiment 6, 7, or 8, wherein the set having the lowest amount of variation is used in calculating a statistically determined estimate of the center of rotation.

[0312] Additional Example 10: The display system of any of Additional Examples 3-9, wherein the parameter comprises a distance from the center of curvature of the cornea to the center of rotation.

[0313] Additional Example 11: The display system of any of Additional Examples 3-9, wherein the parameter comprises a distance along an optical axis from the center of curvature of the cornea to the center of rotation.

[0314] Additional Example 12: The display system of any of the additional embodiments above, wherein the variance comprises a variance and / or a standard deviation.

[0315] Additional Example 13: The display system of any of the additional examples above, further comprising one or more light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes, wherein the inward-facing imaging system forms an image of the eyes using light from the one or more light sources.

[0316] Additional Example 14: 14. The display system of additional example 13, wherein the one or more light sources comprise at least two light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes.

[0317] Additional Example 15: 15. The display system of additional example 13 or 14, wherein the one or more light sources comprise infrared light emitters.

[0318] Additional Example 16: A display system as described in any of additional examples 13-15, wherein the one or more light sources form one or more flashes of light on the eye, and the processing electronics is configured to determine the location of the cornea based on the one or more flashes of light.

[0319] Additional Example 17: A display system as described in any of additional examples 13-16, wherein the cornea has a corneal sphere associated therewith, the cornea having a center of curvature, and the processing electronics is configured to determine the location of the center of curvature of the corneal sphere.

[0320] Additional Example 18: The display system of additional example 17, wherein the cornea has a corneal sphere associated therewith, the cornea having a center of curvature, and the processing electronics is configured to determine the location of the center of curvature of the corneal sphere based on the one or more flashes of light.

[0321] Additional Example 19: The display system of any of the additional embodiments described above, wherein the inward-facing imaging system is configured to image the pupil of the eye.

[0322] Additional Example 20: The display system of any of the additional embodiments described above, wherein the processing electronics is configured to determine a location of a center of the pupil.

[0323] Additional Example 21: The display system of any of the additional embodiments listed above, wherein the processing electronics is configured to determine at least a portion of a boundary between the iris and the pupil.

[0324] Additional Example 22: 22. The display system of additional example embodiment 21, wherein the processing electronics is configured to determine a center of a boundary between the iris and the pupil.

[0325] Additional Example 23: The display system of any of the additional embodiments described above, wherein the processing electronics is configured to determine the location of the center of the pupil in three-dimensional space relative to the center of curvature of the cornea.

[0326] Additional Example 24: The display system of any of the additional embodiments described above, wherein the processing electronics is configured to determine a location and orientation of the optical axis.

[0327] Additional Example 25: 25. The display system of additional example embodiment 24, wherein the processing electronics is configured to determine the location and orientation of the optical axis based on the location of the center of the pupil in three-dimensional space.

[0328] Additional Example 26: The display system of any of the additional examples above, wherein the processing electronics is configured to determine the location and orientation of the optical axis based on the location of the center of the pupil in three-dimensional space relative to the center of curvature of the cornea.

[0329] Additional Example 27: The display system of any of the additional embodiments listed above, wherein the processing electronics is configured to determine a location of a center of rotation of the eye based on a center of curvature of the cornea.

[0330] Additional Example 28: The display system of any of the additional examples described above, wherein the processing electronics is configured to determine a location of a center of rotation of the eye based on the location and orientation of the center of curvature of the cornea and the optical axis.

[0331] Additional Example 29: 29. The display system of claim 28, wherein the processing electronics is configured to determine the location of the center of rotation of the eye by translating a specific distance along the optical axis from the center of curvature of the cornea.

[0332] Additional Example 30: 30. The display system of additional example embodiment 29, wherein the specified distance from the center of curvature to the center of rotation is 4.0 mm to 6.0 mm.

[0333] Additional Example 31: 31. The display system of additional example 29 or 30, wherein the specified distance from the center of curvature to the center of rotation is about 4.7 mm.

[0334] Additional Example 32: The display system of additional example 20 or 30, wherein the processing electronics is configured to determine a specific distance based at least on one or more images of the eye previously acquired using the inward-facing imaging system.

[0335] Additional Example 33: The display system of any of the additional embodiments described above, wherein the processing electronics includes electronics on the frame.

[0336] Additional Example 34: The display system of any of the additional embodiments described above, wherein the processing electronics includes electronics on the frame and electronics located remotely from the frame.

[0337] Additional Example 35: The display system of any of the additional embodiments listed above, wherein the processing electronics includes electronics on the frame and electronics on a belt pack.

[0338] Additional Example 36: A display system as described in any of the above additional examples, wherein at least a portion of the display is transparent and positioned in front of the user's eyes so that when the user wears the head-mounted display, the transparent portion transmits light from a portion of the environment in front of the user and the head-mounted display to the user's eyes and provides a view of a portion of the environment in front of the user and the head-mounted display.

[0339] Additional Example 37: A display system as described in any of the additional examples above, wherein the head-mounted display receives light from a portion of the environment in front of the user's eye at a first amount of divergence and transmits light from the portion of the environment in front of the user's eye to the user's eye with a second amount of divergence that is substantially similar to the first amount of divergence.

[0340] Additional Example 38: The display system of any of the additional examples listed above, wherein the processing electronics is configured to obtain an estimate of the center of rotation by filtering, averaging, applying a Kalman filter to multiple estimated center of rotation positions, or any combination thereof.

[0341] Additional Example 39: The display system of any of the additional embodiments above, wherein the processing electronics is configured to present the virtual image content to the user's eye, rendered as if captured by a camera having an aperture at a determined position of the center of rotation of the user's eye.

[0342] Additional Example 40: The display system of any of the additional embodiments listed above, wherein the processing electronics is configured to use a rendering camera at the center of rotation to render virtual images to be presented to the eyes.

[0343] Additional Example 41:The display system of any of the additional embodiments above, wherein the processing electronics is configured to use a rendering camera configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye.

[0344] Additional Example 42: The display system of any of the additional embodiments above, wherein the processing electronics is configured to use a rendering camera configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture at the center of rotation of the eye.

[0345] Additional Example 43: A display system as described in any of the additional examples above, wherein the processing electronics are configured to use a rendering camera at the center of rotation to render a virtual image to be presented to the eye, and the rendering camera is modeled with an aperture at the center of rotation of the eye.

[0346] Additional Example 44: The display system of any of Additional Examples 1-9, wherein the processing electronics is configured to select a statistically determined estimate of the center of rotation based on the amount of variation during a calibration procedure.

[0347] Additional Example 45: The display system of any of the above or additional embodiments, wherein the processing electronics is configured to obtain an estimate of the center of rotation of the eye based on determining multiple gaze directions of the user's eye over a period of time as the eye rotates, based on images of the eye acquired by the one or more eye tracking cameras.

[0348] Additional Example 46:A display system as described in any of the above examples or additional examples, wherein the processing electronics is configured to determine the gaze direction based on an amount of variation in the shape of one or more of the pupil, iris, or limbus of the user's eye in images acquired using the one or more eye tracking cameras over a period of time as the eye rotates.

[0349] Additional Example 47: A display system as described in any of the above examples or additional examples, wherein the processing electronics is configured to determine an array of positions based on a plurality of spatial locations on images of the user's eyes acquired using the one or more eye tracking cameras.

[0350] Additional Example 48: The display system of any of the above or additional embodiments, wherein the array of positions corresponds to at least a portion of an ellipse.

[0351] Additional Example 49: The display system of any of the above or additional embodiments, wherein the processing electronics is configured to determine the array of positions by fitting a curve to a plurality of spatial locations on the image of the user's eye.

[0352] Additional Example 50: The display system of any of the above or additional embodiments, wherein the curve comprises an ellipse.

[0353] Additional Example 51: The display system of any of the above or additional embodiments, wherein the plurality of spatial locations on the image comprises spatial locations on the limbus of the user's eye in the image.

[0354] Additional Example 52: The display system of any of the above or additional embodiments, wherein the plurality of spatial locations on the image comprises spatial locations on a boundary between the iris and sclera of the user's eye in the image.

[0355] Additional Example 53:A display system as described in any of the above examples or additional examples, wherein the multiple spatial locations on the image include spatial locations on the boundary between the cornea and sclera of the user's eye in the image acquired using the one or more eye tracking cameras.

[0356] Additional Example 54: A display system as described in any of the above examples or additional examples, wherein the processing electronics is configured to determine a plurality of linear paths extending from locations on a first side of the array of locations, through the array of locations, to a second, opposite side of the array of locations.

[0357] Additional Example 55: The display system of any of the above or additional embodiments, wherein the processing electronics is configured to determine a circular region based on the plurality of linear paths, the circular region having a radius R.

[0358] Additional Example 56: The display system of any of the above or additional embodiments, wherein the radius R corresponds to the mean radius of the limbus.

[0359] Additional Example 57: The display system of any of the above or additional embodiments, wherein the radius R corresponds to a measured radius of the limbus of the user's eye.

[0360] Additional Example 58: The display system of any of the above or additional embodiments, wherein the radius R corresponds to the average radius of the pupil.

[0361] Additional Example 59: The display system of any of the above or additional embodiments, wherein the radius R corresponds to a measured radius of a pupil of a user's eye.

[0362] Additional Example 60: The display system of any of the above or additional embodiments, wherein the processing electronics is configured to determine a location and direction of a normal that passes through a central portion of the circular region.

[0363] Additional Example 61: A display system as described in any of the above examples or additional examples, wherein the processing electronics is configured to determine respective locations and directions of multiple normals passing through central portions of the respective circular regions based on multiple images of the eye previously acquired using the one or more eye tracking cameras.

[0364] Additional Example 62: The display system of any of the above or additional embodiments, wherein the processing electronics is configured to determine a location where the normals converge or intersect.

[0365] Additional Example 63: A display system as described in any of the above examples or additional examples, wherein the processing electronics is configured to obtain an estimate of the center of rotation of the user's eye by identifying areas of intersection, convergence, or close proximity of multiple ones of the normals, determined based on images of the user's eye acquired over a period of time while the eye is rotating.

[0366] Additional Example 64: A display system as described in any of the above examples or additional examples, wherein the processing electronics is configured to obtain an estimate of the center of rotation of the user's eye based on the locations and orientations of multiple ones of the multiple normals, determined based on images of the user's eye acquired over a period of time while the eye is rotating.

[0367] Additional Example 65: A display system as described in any of the above or additional embodiments, wherein the location on the first side of the array position corresponds to the origin of a coordinate system of one of the one or more eye tracking cameras.

[0368] Additional Example 66: The display system of any of the above or additional examples, wherein the processing electronics is configured to obtain an estimate of the center of rotation by filtering, averaging, applying a Kalman filter to multiple estimated center of rotation positions, or any combination thereof. Any of the above embodiments or additional embodiments can be combined. In addition, any of the above embodiments or additional embodiments can be integrated with a head-mounted display. In addition, any of the above embodiments or additional embodiments can be implemented using a single depth plane and / or one or more variable depth planes (e.g., one or more elements with variable focusing power that provide accommodation cues that vary over time). Additionally, disclosed herein are devices and methods for determining various values, parameters, such as, but not limited to, anatomical, optical, and geometric features, locations, and orientations. Examples of such parameters include, but are not limited to, the center of rotation of the eye, the center of curvature of the cornea, the center of the pupil, the pupil boundary, the center of the iris, the iris boundary, the limbus boundary, the optical axis of the eye, the visual axis of the eye, and the center of gaze. Determinations of such values, parameters, etc., as listed herein, include estimates thereof and do not necessarily correspond precisely to actual values. For example, determinations of the center of rotation of the eye, the center of curvature of the cornea, the center or boundary of the pupil or iris, the limbus boundary, the optical axis of the eye, the visual axis of the eye, the center of gaze, etc., may be estimates, approximations, or close values ​​that are not identical to actual (e.g., anatomical, optical, or geometric) values ​​or parameters. In some cases, for example, root-mean-square estimation techniques are used to obtain estimates of such values. As an example, certain techniques described herein relate to identifying locations or points where rays or vectors intersect. However, such rays or vectors may not intersect. In this example, the locations or points may be estimated. For example, the locations or points may be determined based on root mean square or other estimation techniques (e.g., the locations or points may be estimated to be close or nearest to the rays or vectors). Other processes may also be used to estimate approximate values ​​or otherwise provide values ​​that may not correspond to actual values. Thus, the terms “determine” and “estimate” or “determined” and “estimated” are used interchangeably herein. References to such determined values ​​may therefore include estimates, approximations, or values ​​near the actual values. Thus, references to determining a parameter or value above or elsewhere herein should not be limited to the exact actual value, but may include estimates, approximations, or values ​​near the actual value.

[0369] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this summary nor the following detailed description purports to define or limit the scope of the inventive subject matter. The present invention provides, for example, the following. (Item 1) 1. A display system configured to project light onto a user's eye and display virtual image content within a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis, the optical axis extending through the lens, the pupil, and the cornea, the display system comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project light into the user's eyes with at least one of different amounts of divergence and collimation to display virtual image content in the user's field of view, whereby the displayed virtual image content appears to arise from different depths and at different time periods; one or more eye-tracking cameras configured to image the eyes of the user; processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to obtain an estimate of a center of rotation of the eye based on images of the eye acquired using the one or more eye-tracking cameras; A display system comprising: (Item 2) Item 10. The display system of item 1, further comprising one or more light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more eye tracking cameras forming images of the eyes using light from the one or more light sources. (Item 3) 3. The display system of claim 2, wherein the one or more light sources comprise at least two light sources, the at least two light sources being positioned on the frame relative to the user's eyes and illuminating the user's eyes. (Item 4) Item 4. The display system of item 3, wherein the one or more light sources comprise infrared light emitters. (Item 5) Item 3. The display system of item 2, wherein the one or more light sources form one or more flashes of light on the eye, and the processing electronics is configured to determine the location of the cornea based on the one or more flashes of light. (Item 6) Item 6. The display system of item 5, wherein the cornea has a corneal sphere associated therewith, the corneal sphere having a center of curvature, and the processing electronics is configured to determine the location of the center of curvature of the corneal sphere based on the one or more flashes of light. (Item 7) Item 10. The display system of item 1, wherein the cornea has a corneal sphere associated therewith, the corneal sphere having a center of curvature, and the processing electronics is configured to determine the location of the center of curvature of the corneal sphere. (Item 8) Item 10. The display system of item 1, wherein the one or more eye-tracking cameras are configured to image the pupil of the eye. (Item 9) Item 10. The display system of item 1, wherein the processing electronics is configured to determine the location of the center of the pupil. (Item 10) Item 10. The display system of item 1, wherein the processing electronics is configured to determine at least a portion of a boundary between the iris and the pupil. (Item 11) Item 11. The display system of item 10, wherein the processing electronics is configured to determine a center of a boundary between the iris and the pupil. (Item 12) Item 10. The display system of item 1, wherein the processing electronics is configured to determine the location of the center of the pupil in three-dimensional space relative to the center of curvature of the cornea. (Item 13) Item 14. The display system of item 12, wherein the processing electronics is configured to determine the location and orientation of the optical axis based on the location of the center of the pupil in three-dimensional space. Item 10. The display system of item 1, wherein the processing electronics is configured to determine the location and orientation of the optical axis. (Item 15) Item 15. The display system of item 14, wherein the processing electronics is configured to determine the location and orientation of the optical axis based on the location of the center of the pupil in three-dimensional space relative to the center of curvature of the cornea. (Item 16) Item 10. The display system of item 1, wherein the processing electronics is configured to determine a location of a center of rotation of the eye based on a center of curvature of the cornea. (Item 17) Item 10. The display system of item 1, wherein the processing electronics is configured to determine the location of the eye's center of rotation based on the location and orientation of the corneal center of curvature and the optical axis. (Item 18) Item 18. The display system of item 17, wherein the processing electronics is configured to determine the location of the eye's center of rotation by translating a specific distance along the optical axis from the corneal center of curvature. (Item 19) Item 19. The display system of item 18, wherein the specific distance from the center of curvature to the center of rotation is 4.0 mm to 6.0 mm. (Item 20) Item 19. The display system of item 18, wherein the specific distance from the center of curvature to the center of rotation is approximately 4.7 mm. (Item 21) Item 19. The display system of item 18, wherein the particular distance is fixed. (Item 22) Item 19. The display system of item 18, wherein the processing electronics is configured to determine the particular distance based at least on one or more images of the eye previously acquired using the one or more eye tracking cameras. (Item 23) Item 10. The display system of item 1, wherein the processing electronics is configured to determine a location and orientation of a visual axis offset from the optical axis based on the location and orientation of the optical axis. (Item 24) Item 10. The display system of item 1, wherein the processing electronics is configured to determine the location and orientation of a viewing axis based on an angular rotation relative to the optical axis. (Item 25) Item 10. The display system of item 1, wherein the processing electronics is configured to determine the location and orientation of the visual axis based on an angular rotation of 4.0° to 6.5° relative to the optical axis. (Item 26) Item 10. The display system of item 1, wherein the processing electronics is configured to determine the location and orientation of the viewing axis based on an angular rotation of approximately 5.2° relative to the optical axis. (Item 27) Item 10. The display system of item 1, wherein the processing electronics is configured to determine the location and orientation of the visual axis based at least on one or more images of the eye previously acquired using the one or more eye tracking cameras. (Item 28) Item 10. The display system of item 1, wherein the processing electronics is configured to determine the center of rotation of the eye based on multiple determinations of the location of the optical axis or visual axis over a period of time during which the eye is rotating. (Item 29) Item 10. The display system of item 1, wherein the processing electronics is configured to determine the center of rotation of the eye by identifying an area of ​​intersection of multiple determinations of the location of the optical axis or visual axis over a period of time during which the eye is rotating. (Item 30) Item 10. The display system of claim 1, wherein the processing electronics is configured to determine the user's convergence and divergence distances at which the user's left and right eyes are gazed based on determining the location and orientation of the optical axes relative to the user's left and right eyes. (Item 31) Item 10. The display system of claim 1, wherein the processing electronics is configured to determine the user's convergence and divergence distances at which the user's left and right eyes are gazed based on determining the location and orientation of the visual axes relative to the user's left and right eyes. (Item 32) Item 10. The display system of claim 1, wherein the processing electronics is configured to determine the convergence and divergence distances at which the user's left and right eyes are gazed based on identifying regions of intersection of visual axes for the user's left and right eyes. (Item 33) Item 10. The display system of claim 1, wherein the processing electronics is configured to determine the convergence and divergence distances at which the user's left and right eyes are gazed by projecting the visual axes of the left and right eyes onto a horizontal plane and identifying an area of ​​intersection of the projections of the visual axes of the left and right eyes onto the horizontal plane. (Item 34) Item 10. The display system of item 1, wherein the processing electronics is configured to determine a relative amount of at least one of divergence and collimation for projecting image content based on a determination of the user's convergence-divergence distance. (Item 35) Item 1. The display system of item 1, wherein the processing electronics includes electronics on the frame. (Item 36) Item 1. The display system of item 1, wherein the processing electronics includes electronics on the frame and electronics located remotely from the frame. (Item 37) Item 1. The display system of item 1, wherein the processing electronics includes electronics on the frame and electronics on a belt pack. (Item 38) Item 1. A display system as described in item 1, wherein at least a portion of the display is transparent and positioned in front of the user's eyes so that when the user wears the head-mounted display, the transparent portion transmits light from a portion of the environment in front of the user and the head-mounted display to the user's eyes and provides a view of a portion of the environment in front of the user and the head-mounted display. (Item 39) Item 1. The display system of item 1, wherein the head-mounted display receives light from a portion of the environment in front of the user at a first divergence amount and transmits light from the portion of the environment in front of the user to the user's eyes with a second divergence amount that is substantially similar to the first divergence amount. (Item 40) Item 10. The display system of item 1, wherein the processing electronics is configured to obtain the estimate of the center of rotation by filtering, averaging, applying a Kalman filter to multiple estimated center of rotation positions, or any combination thereof. (Item 41) Item 10. The display system of claim 1, wherein the processing electronics are configured to present the virtual image content to the user's eyes, rendered as if captured by a camera having an aperture at a determined position of the center of rotation of the user's eyes. (Item 42) Item 10. The display system of item 1, wherein the processing electronics is configured to use a rendering camera at the center of rotation to render a virtual image to be presented to the eye. (Item 43) Item 10. The display system of item 1, wherein the processing electronics is configured to use a rendering camera configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye. (Item 44) Item 10. The display system of item 1, wherein the processing electronics is configured to use a rendering camera configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture at the center of rotation of the eye. (Item 45) Item 1. The display system of item 1, wherein the processing electronics is configured to use a rendering camera at the center of rotation to render a virtual image to be presented to the eye, the rendering camera being modeled with an aperture at the center of rotation of the eye. (Item 46) Item 10. The display system of claim 1, wherein the processing electronics is configured to obtain an estimate of the center of rotation of the eye based on determining multiple gaze directions of the user's eye over a period of time during which the eye is rotating, based on images of the eye acquired by the one or more eye tracking cameras. (Item 47) Item 47. The display system of item 46, wherein the processing electronics is configured to determine the gaze direction based on an amount of variation in the shape of one or more of the pupil, iris, or limbus of the user's eye in images acquired using the one or more eye tracking cameras over a period of time during which the eye is rotating. (Item 48) Item 10. The display system of item 1, wherein the processing electronics is configured to determine an array of positions based on a plurality of spatial locations on an image of the user's eye acquired using the one or more eye tracking cameras. (Item 49) Item 49. The display system of item 48, wherein the array of positions corresponds to at least a portion of an ellipse. (Item 50) Item 49. The display system of item 48, wherein the processing electronics is configured to determine the array of positions by fitting a curve to a plurality of spatial locations on the image of the user's eye. (Item 51) Item 51. The display system of item 50, wherein the curve comprises an ellipse. (Item 52) Item 50. The display system of item 49, wherein the plurality of spatial locations on the image comprises spatial locations on the limbus of the user's eye in the image. (Item 53) Item 50. The display system of item 49, wherein the plurality of spatial locations on the image comprises spatial locations on a boundary between the iris and sclera of the user's eye in the image. (Item 54) Item 50. The display system of item 49, wherein the plurality of spatial locations on the image comprises spatial locations on the boundary between the cornea and the sclera of the user's eye in the image acquired using the one or more eye tracking cameras. (Item 55) Item 50. The display system of item 49, wherein the processing electronics is configured to determine a plurality of linear paths extending from locations on a first side of the array of locations, through the array of locations, to a second, opposite side of the array of locations. (Item 56) Item 56. The display system of item 55, wherein the processing electronics is configured to determine a circular region based on the plurality of linear paths, the circular region having a radius R. (Item 57) Item 57. The display system of item 56, wherein the radius R corresponds to the mean radius of the limbus. (Item 58) Item 57. The display system of item 56, wherein the radius R corresponds to the measured radius of the limbus of the user's eye. (Item 59) Item 60. The display system of item 56, wherein the radius R corresponds to the average radius of the pupil. Item 57. The display system of item 56, wherein the radius R corresponds to a measured radius of the pupil of the user's eye. (Item 61) Item 57. The display system of item 56, wherein the processing electronics is configured to determine the location and direction of a normal passing through a central portion of the circular region. (Item 62) Item 57. The display system of item 56, wherein the processing electronics is configured to determine individual locations and directions of multiple normals passing through central portions of the individual circular regions based on multiple images of the eye previously acquired using the one or more eye tracking cameras. (Item 63) Item 63. The display system of item 62, wherein the processing electronics is configured to determine locations where the multiple normals converge or intersect. (Item 64) Item 63. The display system of item 62, wherein the processing electronics is configured to obtain an estimate of the center of rotation of the user's eye by identifying an area of ​​intersection of multiple of the normals determined based on images of the user's eye acquired over a period of time while the eye is rotating. (Item 65) Item 63. The display system of item 62, wherein the processing electronics is configured to obtain an estimate of the center of rotation of the user's eye based on the locations and orientations of multiple ones of the multiple normals determined based on images of the user's eye acquired over a period of time while the eye is rotating. (Item 66) Item 56. A display system as described in Item 55, wherein the location on the first side of the array position corresponds to the origin of a coordinate system of one of the one or more eye-tracking cameras. (Item 67) 1. A display system configured to project light onto a user's eye and display virtual image content within a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis, the optical axis extending through the lens, the pupil, and the cornea, the display system comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project light into the user's eyes with at least one of different amounts of divergence and collimation to display virtual image content in the user's field of view, whereby the displayed virtual image content appears to arise from different depths and at different time periods; one or more eye-tracking cameras configured to image the eyes of the user; processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to obtain a position estimate of a center of gaze of the eye based on images of the eye acquired using the one or more eye-tracking cameras, the center of gaze being estimated to be proximate to a pupil of the eye or between the cornea and the pupil of the eye; and Equipped with The display system, wherein the processing electronics is configured to present the virtual image content rendered by a rendering camera located at the viewpoint center to the user's eyes. (Item 68) 1. A method of rendering virtual image content in a display system configured to project light onto a user's eye and display the virtual image content within a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis, the optical axis extending through the lens, the pupil, and the cornea, the method comprising: determining a position of a center of rotation of the eye using one or more eye tracking cameras configured to image the user's eye and track movement of the eye; Rendering virtual image content using a rendering engine with a rendering camera at a center of rotation of the eye, the rendering camera configured to render a virtual image to be presented to the eye; using a head-mounted display to project light into the user's eyes at different amounts of divergence so that the virtual image content appears to arise from different depths and at different time periods, and displaying the rendered virtual image content in the user's field of view; A method comprising: (Item 69) 1. A display system configured to project light onto a user's eye and display virtual image content within a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis, the optical axis extending through the lens, the pupil, and the cornea, the display system comprising: a frame configured to be supported on the user's head; a head mounted display disposed on the frame, the display configured to project light at different amounts of divergence into the user's eyes and display virtual image content in the user's field of view, whereby the displayed virtual image content appears to arise from different depths and for different time periods, the head mounted display configured to project light having a first amount of divergence into the user's eyes in a first time period and to project light having a second amount of divergence into the user's eyes in a second time period, the first amount of divergence being different from the second amount of divergence; one or more eye-tracking cameras configured to image the eyes of the user; processing electronics in communication with the display and the one or more eye tracking cameras, the processing electronics configured to: obtain an estimate of a center of rotation of the eye based on images of the eye acquired with the one or more eye tracking cameras; obtain an estimate of a vergence distance of the user based on images of the eye acquired with the one or more eye tracking cameras; and shift from projecting light into the user's eye at the first divergence amount to projecting light into the user's eye at the second divergence amount based on the estimated vergence distance of the user; A display system comprising: (Item 70) 1. A display system configured to project light onto a user's left and right eyes to display virtual image content within the user's field of view, each of the eyes having a cornea, an iris, a pupil, a lens, a retina, and an optical axis, the optical axis extending through the lens, pupil, and cornea, the display system comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project light into the user's left and right eyes with different amounts of divergence and at least one of collimation and display virtual image content in the user's field of view, whereby the displayed virtual image content appears to originate from different distances and at different time periods from the user's left and right eyes; a first eye-tracking camera configured to image the left eye of the user; a second eye-tracking camera configured to image the user's right eye; and processing electronics in communication with the display and the first and second eye tracking cameras, the processing electronics configured to obtain an estimate of an interpupillary distance between the user's left and right eyes based on the left and right eye images acquired using the first and second eye tracking cameras; and A display system comprising: (Item 71) 1. A method of rendering virtual image content in a display system configured to project light to a left and right eye of a user and display the virtual image content within a field of view of the user, wherein the eyes each have a cornea, an iris, a pupil, a lens, a retina, and an optical axis, the optical axis extending through the lens, pupil, and cornea, the method comprising: determining a position of a center of rotation of the left eye and a position of a center of rotation of the right eye using one or more eye tracking cameras configured to image the user's eyes and track movement of the eyes; using processing electronics in communication with the one or more eye tracking cameras to estimate the user's interpupillary distance based on the determined positions of the centers of rotation of the left and right eyes; determining a current left eye pose and a current right eye pose using the one or more eye tracking cameras; using the processing electronics to estimate the user's current convergence-divergence distance by comparing the estimated interpupillary distance with the determined current left eye posture and the determined current right eye posture; A method comprising: (Item 72) 1. A display system configured to project light onto a user's eye and display virtual image content within a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis, the optical axis extending through the lens, the pupil, and the cornea, the display system comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project light into the user's eyes with at least one of different amounts of divergence and collimation to display virtual image content in the user's field of view, whereby the displayed virtual image content appears to arise from different depths and at different time periods; one or more eye-tracking cameras configured to image the eyes of the user; processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to obtain a position estimate of a center of rotation of the eye based on images of the eye acquired using the one or more eye-tracking cameras, and to obtain a direction estimate of an optical axis of the eye based on the images; Equipped with The processing electronics are configured to present the virtual image content to the user's eye, rendered as if captured by a camera positioned along the optical axis and having an aperture spaced from the estimated position of the eye's center of rotation so as to be 6.0 mm to 13.0 mm away from the retina in a direction. (Item 73) 1. A display system configured to project light onto a user's eye and display virtual image content within a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis, the optical axis extending through the lens, the pupil, and the cornea, the display system comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project light into the user's eyes with at least one of different amounts of divergence and collimation to display virtual image content in the user's field of view, whereby the displayed virtual image content appears to arise from different depths and at different time periods; one or more eye-tracking cameras configured to image the eyes of the user; processing electronics in communication with the display and the one or more eye-tracking cameras; Equipped with A display system wherein the processing electronics are configured to present the virtual image content to the user's eye rendered by a rendering camera positioned at the pupil of the eye or between the pupil and the cornea of ​​the eye. (Item 74) 1. A display system configured to project light onto a user's eye and display virtual image content within a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis, the optical axis extending through the lens, the pupil, and the cornea, the display system comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project light into the user's eyes and display virtual image content in the user's field of view, at least a portion of the display being transparent and positioned in a location in front of the user's eyes such that, when the user wears the frame, the transparent portion transmits light from a portion of the environment in front of the user and the head-mounted display to the user's eyes and provides a view of the portion of the environment in front of the user and the head-mounted display; an inward-facing imaging system configured to image the user's eye; processing electronics in communication with the inward-facing imaging system, the processing electronics configured to obtain an estimate of a center of rotation of the eye based on a plurality of images of the eye acquired with the inward-facing imaging system, the processing electronics configured to determine a variance in calculated values ​​of the center of rotation and select a statistically determined estimate of the center of rotation based on the variance; A display system comprising: (Item 75) 1. A method of rendering virtual image content in a display system configured to project light onto a user's eye and display the virtual image content within a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis, the optical axis extending through the lens, the pupil, and the cornea, the method comprising: using one or more eye cameras configured to image the user's eye, determining a position based on imaging of the eye using the one or more cameras; using a rendering engine to render virtual image content using a rendering camera at a location based on the determined position, the rendering camera configured to render a virtual image to be presented to the eye; projecting light into the user's eye using a head mounted display to display the rendered virtual image content in the user's field of view; A method comprising: (Item 76) 1. A display system configured to project light onto a user's eye and display virtual image content within a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis, the optical axis extending through the lens, the pupil, and the cornea, the display system comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project light into the user's eye and display virtual image content in the user's field of view; one or more cameras configured to image the user's eyes; processing electronics in communication with the display and the one or more cameras, the processing electronics configured to obtain a position of the eye based on images of the eye obtained using the one or more cameras; and Equipped with The display system, wherein the processing electronics is configured to present the virtual image content to the user's eyes rendered by a rendering camera located at a location based on the determined position. (Item 77) 1. A display system configured to project light onto a user's eye and display virtual image content within the user's field of view, the eye having a cornea, an iris, a pupil, a lens, and a retina, the display system comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project light into the user's eyes with at least one of different amounts of divergence and collimation to display virtual image content in the user's field of view, whereby the displayed virtual image content appears to arise from different depths and at different time periods; one or more eye-tracking cameras configured to image the eyes of the user; processing electronics in communication with the display and the one or more eye tracking cameras, the processing electronics comprising: obtaining a position and orientation estimate of an optical axis extending through a lens, pupil, and cornea of ​​the user's eye based on images of the eye obtained using the one or more eye tracking cameras; Identifying a specific location along an axis in a rendering space that is aligned with the optical axis of the eye; presenting the virtual image content to the user's eye as rendered by a virtual rendering camera located at the particular location within a rendering space; and processing electronics configured to perform A display system comprising: [Brief explanation of the drawings]

[0370] [Figure 1] FIG. 1 depicts an illustration of a mixed reality scenario with a virtual reality object and a physical object viewed by a person.

[0371] [Figure 2]FIG. 2 illustrates diagrammatically an example of a wearable system.

[0372] [Figure 3] FIG. 3 diagrammatically illustrates example components of a wearable system.

[0373] [Figure 4] FIG. 4 diagrammatically illustrates an example of a waveguide stack of a wearable device for outputting image information to a user.

[0374] [Figure 5] FIG. 5 illustrates a schematic example of an eye.

[0375] [Figure 5A] FIG. 5A diagrammatically illustrates an exemplary coordinate system for determining the eye posture of the eye.

[0376] [Figure 6] FIG. 6 is a schematic diagram of a wearable system including an eye tracking system.

[0377] [Figure 7A] FIG. 7A is a block diagram of a wearable system that may include an eye tracking system.

[0378] [Figure 7B] FIG. 7B is a block diagram of a rendering controller in a wearable system.

[0379] [Figure 8A] FIG. 8A is a schematic diagram of the eye showing the spherical cornea of ​​the eye.

[0380] [Figure 8B] FIG. 8B illustrates an exemplary corneal phosphene detected by an eye-tracking camera.

[0381] [Figure 8C]8C-8E illustrate exemplary steps for locating a user's corneal center using an eye tracking module in a wearable system. [Figure 8D] 8C-8E illustrate exemplary steps for locating a user's corneal center using an eye tracking module in a wearable system. [Figure 8E] 8C-8E illustrate exemplary steps for locating a user's corneal center using an eye tracking module in a wearable system.

[0382] [Figure 9A] 9A-9C illustrate an exemplary normalization of the coordinate system of the eye-tracking image. [Figure 9B] 9A-9C illustrate an exemplary normalization of the coordinate system of the eye-tracking image. [Figure 9C] 9A-9C illustrate an exemplary normalization of the coordinate system of the eye-tracking image.

[0383] [Figure 9D] 9D-9G illustrate exemplary steps for locating a user's pupil center using an eye tracking module in a wearable system. [Figure 9E] 9D-9G illustrate exemplary steps for locating a user's pupil center using an eye tracking module in a wearable system. [Figure 9F] 9D-9G illustrate exemplary steps for locating a user's pupil center using an eye tracking module in a wearable system. [Figure 9G] 9D-9G illustrate exemplary steps for locating a user's pupil center using an eye tracking module in a wearable system.

[0384] [Figure 9H] FIG. 9H illustrates exemplary calculated locations of the pupil center resulting from both considering corneal refraction and not considering corneal refraction.

[0385] [Figure 9I] 9I-9M illustrate exemplary experimental variations in the calculated center of rotation of the eye based on different calculated pupil centers. [Figure 9J] 9I-9M illustrate exemplary experimental variations in the calculated center of rotation of the eye based on different calculated pupil centers. [Figure 9K] 9I-9M illustrate exemplary experimental variations in the calculated center of rotation of the eye based on different calculated pupil centers. [Figure 9L] 9I-9M illustrate exemplary experimental variations in the calculated center of rotation of the eye based on different calculated pupil centers. [Figure 9M] 9I-9M illustrate exemplary experimental variations in the calculated center of rotation of the eye based on different calculated pupil centers.

[0386] [Figure 10] FIG. 10 illustrates an example of an eye, including the optical and visual axes of the eye and the center of rotation of the eye.

[0387] [Figure 11] FIG. 11 is a process flow diagram of an example method for using eye tracking to provide feedback regarding alignment within a wearable device when rendering content.

[0388] [Figure 12] FIG. 12 is a set of example graphs illustrating how a wearable system can switch depth planes in response to a user's eye movements.

[0389] [Figure 13] FIG. 13 depicts a mixed reality system in which a virtual object may be depth-discretized into one or more depth planes.

[0390] [Figure 14A] 14A and 14B illustrate an example in which a mixed reality system displays virtual objects using a single depth plane. [Figure 14B] 14A and 14B illustrate an example in which a mixed reality system displays virtual objects using a single depth plane.

[0391] [Figure 14C] 14C and 14D illustrate an example in which a mixed reality system uses two adjacent depth planes to display a virtual object and generate accommodation cues between the two adjacent depth planes. [Figure 14D] 14C and 14D illustrate an example in which a mixed reality system uses two adjacent depth planes to display a virtual object and generate accommodation cues between the two adjacent depth planes.

[0392] [Figure 14E] 14E and 14F illustrate an example in which a mixed reality system uses two or more depth planes to display virtual objects and generate two or more accommodation cues simultaneously. [Figure 14F] 14E and 14F illustrate an example in which a mixed reality system uses two or more depth planes to display virtual objects and generate two or more accommodation cues simultaneously.

[0393] [Figure 15A] FIG. 15A illustrates an example in which a mixed reality system displays a virtual object in the manner of FIGS. 14A and 14B in the presence of viewpoint center misalignment.

[0394] [Figure 15B] FIG. 15B illustrates an example in which a mixed reality system displays a virtual object in the manner of FIGS. 14C and 14D in the presence of viewpoint center misalignment.

[0395] [Figure 15C] FIG. 15C illustrates an example in which a mixed reality system displays a virtual object in the manner of FIGS. 14E and 14F in the presence of viewpoint center misalignment.

[0396] [Figure 16A] 16A and 16B illustrate an example in which the pinhole of the rendering camera is aligned with the eye's center of view or approximately with the eye's pupil. [Figure 16B] 16A and 16B illustrate an example in which the pinhole of the rendering camera is aligned with the eye's center of view or approximately with the eye's pupil.

[0397] [Figure 16C] FIG. 16C illustrates another example in which the pinhole of the rendering camera is aligned with the eye's center of view or approximately with the eye's pupil.

[0398] [Figure 17A] 17A and 17B illustrate an example in which the pinhole of the rendering camera is aligned with the center of rotation of the eye. [Figure 17B] 17A and 17B illustrate an example in which the pinhole of the rendering camera is aligned with the center of rotation of the eye.

[0399] [Figure 17C] FIG. 17C illustrates another example where the pinhole of the rendering camera is aligned with the center of rotation of the eye.

[0400] [Figure 18A] 18A and 18B are a set of example graphs illustrating eye tracking data relative to a user's center of gaze. [Figure 18B] 18A and 18B are a set of example graphs illustrating eye tracking data relative to a user's center of gaze.

[0401] [Figure 18C] FIG. 18C is an exemplary graph illustrating how the rendering camera position can vary as a function of eye tracking data.

[0402] [Figure 18D]FIG. 18D is a schematic diagram of a user's eye illustrating various rendering camera positions that may be utilized as a function of eye tracking data.

[0403] [Figure 19] 19 illustrates a graphical representation of the use of an ellipse projected onto an image of a user's eye (e.g., limbus) used to determine an estimate of CoR in accordance with various techniques described herein. Rays may be traced through the projected ellipse to form a cone to which a circle is fitted. In some implementations, a normal through one or more circles may be used to estimate CoR.

[0404] [Figure 20] FIG. 20 is a flowchart of an exemplary process for determining the center of rotation of an eye based on the limbus of the eye.

[0405] [Figure 21A] 21A illustrates a first ellipse (hereinafter referred to as the "projected ellipse") projected onto an image of a user's limbus, determined based on the user's first line of sight. Light rays passing through the projected ellipse are shown, forming a cone to which a circle is fitted.

[0406] [Figure 21B] 21B illustrates a second projected ellipse determined based on a second line of sight of the user's eye. A second cone and a circle fitted thereto are also shown.

[0407] [Figure 21C] 21C illustrates a third projected ellipse determined based on a third line of sight of the user's eye. A third cone and a circle fitted thereto are also shown.

[0408] [Figure 21D] FIG. 21D illustrates the step of determining the center of rotation based on a circular shape obtained using a projected ellipse as described above.

[0409] [Figure 22] FIG. 22 illustrates an optical system including two point light sources, an aperture, a lens, and a projection screen.

[0410] [Figure 23A] FIG. 23A illustrates an embodiment of the optical system in a first stage.

[0411] [Figure 23B] FIG. 23B illustrates the optical system in the second stage.

[0412] [Figure 24A] FIG. 24A illustrates another embodiment of the optical system in the first stage.

[0413] [Figure 24B] FIG. 24B illustrates the optical system in the second stage.

[0414] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0415] Reference will now be made to the drawings, in which like reference numerals refer to like parts throughout. Unless otherwise indicated, the drawings are schematic and are not necessarily drawn to scale.

[0416] Example of a 3D display for a wearable system A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present 2D or 3D virtual images to a user. The images may be still images, frames of video, or videos, in combination or the like. At least a portion of the wearable system can be implemented on a wearable device, which may present a VR, AR, or MR environment, alone or in combination, for user interaction. A wearable device can be used synonymously with an AR device (ARD). Additionally, for purposes of this disclosure, the term "AR" is used synonymously with the term "MR."

[0417] Figure 1 depicts an illustration of a mixed reality scenario involving a virtual reality object and a physical object viewed by a person. In Figure 1, an MR scene 100 is depicted in which a user of the MR technology sees a real-world park-like setting 110 featuring people, trees, a building in the background, and a concrete platform 120. In addition to these items, the user of the MR technology also perceives as "seeing" a robotic figure 130 standing on the real-world platform 120 and a flying, cartoon-like avatar character 140 that appears to be an anthropomorphic bumblebee, although these elements do not exist in the real world.

[0418] In order for a 3D display to produce a true depth sensation, and more specifically, a simulated sensation of surface depth, it may be desirable for the display to generate, for each point in its field of view, an accommodation response that corresponds to that point's virtual depth. If the accommodation response to a display point does not correspond to that point's virtual depth as determined by convergence and stereoscopic binocular depth cues, the human eye may experience accommodation conflict, resulting in unstable imaging, adverse eye strain, headaches, and, in the absence of accommodative information, a near-complete lack of surface depth.

[0419] VR, AR, and MR experiences can be provided by a display system having a display that provides a viewer with images corresponding to multiple depth planes. The images may be different for each depth plane (e.g., providing slightly different presentations of a scene or object) and may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features of a scene located on different depth planes, or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a believable perception of depth.

[0420] FIG. 2 illustrates an example of a wearable system 200, which can be configured to provide an AR / VR / MR scene. The wearable system 200 may also be referred to as an AR system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems for supporting the functionality of the display 220. The display 220 may be coupled to a frame 230, which is wearable by a user, wearer, or viewer 210. The display 220 can be positioned directly in front of the eyes of the user 210. The display 220 can present AR / VR / MR content to the user. The display 220 can comprise a head-mounted display (HMD) worn on the user's head.

[0421] In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent the user's other ear canal to provide stereo / shapeable sound control). The display 220 can include an audio sensor (e.g., a microphone) 232 to detect audio streams from the environment and capture ambient sounds. In some embodiments, one or more other audio sensors, not shown, are positioned to provide stereo sound reception. The stereo sound reception can be used to determine the location of a sound source. The wearable system 200 can perform voice or speech recognition on the audio stream.

[0422] The wearable system 200 may include an outward-facing imaging system 464 (shown in FIG. 4 ) that observes the world in the user's surrounding environment. The wearable system 200 may also include an inward-facing imaging system 462 (shown in FIG. 4 ) that may track the user's eye movements. The inward-facing imaging system may track either one eye's movements or both eyes' movements. The inward-facing imaging system 462 may be mounted to the frame 230 and may be in electrical communication with a processing module 260 or 270 that may process image information obtained by the inward-facing imaging system and determine, for example, the pupil diameter or orientation of the user's 210 eyes, eye movement, or eye posture. The inward-facing imaging system 462 may include one or more cameras. For example, at least one camera may be used to image each eye. Images obtained by the cameras may be used to determine pupil size or eye posture for each eye separately, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye.

[0423] As an example, the wearable system 200 can obtain an image of the user's posture using an outward-facing imaging system 464 or an inward-facing imaging system 462. The image may be a still image, a frame from a video, or a video.

[0424] The display 220 can be operably coupled (250) to a local data processing module 260, which can be mounted in a variety of configurations, such as fixedly attached to the frame 230, by wired or wireless connection, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0425] The local processing and data module 260 may comprise a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data may include a) data captured from sensors (e.g., that may be operatively coupled to the frame 230 or otherwise attached to the user 210), such as an image capture device (e.g., a camera in an inward-facing or outward-facing imaging system), audio sensors (e.g., a microphone), an inertial measurement unit (IMU), an accelerometer, a compass, a global positioning system (GPS) unit, a wireless device, or a gyroscope, or b) data obtained or processed using the remote processing module 270 or the remote data repository 280, possibly for passing to the display 220 after processing or retrieval. The local processing and data module 260 may be operably coupled to a remote processing module 270 or a remote data repository 280 by a communication link 262 or 264, such as via a wired or wireless communication link, so that these remote modules are available as resources to the local processing and data module 260. In addition, the remote processing module 280 and the remote data repository 280 may be operably coupled to each other.

[0426] In some embodiments, remote processing module 270 may comprise one or more processors configured to analyze and process data or image information. In some embodiments, remote data repository 280 may comprise a digital data storage facility, which may be available through the Internet or other networking configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from the remote module.

[0427] Example Components of a Wearable System FIG. 3 diagrammatically illustrates example components of a wearable system. FIG. 3 shows a wearable system 200, which may include a display 220 and a frame 230. A blowup 202 diagrammatically illustrates various components of the wearable system 200. In some implementations, one or more of the components illustrated in FIG. 3 may be part of the display 220. The various components, alone or in combination, may collect various data associated with a user of the wearable system 200 or the user's environment (e.g., auditory or visual data, etc.). It should be understood that other embodiments may have additional or fewer components, depending on the application for which the wearable system is used. Note that FIG. 3 provides a basic idea of ​​some of the various components and the types of data that may be collected, analyzed, and stored through the wearable system.

[0428] FIG. 3 shows an exemplary wearable system 200, which may include a display 220. The display 220 may include a display lens 226 that may be mounted to a housing or frame 230 that corresponds to the user's head. The display lens 226 may include one or more transparent mirrors positioned by the housing 230 in front of the user's eyes 302, 304 and may be configured to bounce projected light 338 into the eyes 302, 304, facilitating beam shaping while also allowing transmission of at least some light from the local environment. The wavefront of the projected light beam 338 may be bent or focused to match a desired focal length of the projected light. As shown, two wide-field machine vision cameras 316 (also referred to as world cameras) are coupled to the housing 230 and can image the environment around the user. These cameras 316 may be dual-capture visible / invisible (e.g., infrared) light cameras. Camera 316 may be part of the outward-facing imaging system 464 shown in Figure 4. Images acquired by world camera 316 may be processed by pose processor 336. For example, pose processor 336 may implement one or more object recognizers 708 (e.g., shown in Figure 7) to identify the pose of the user or another person in the user's environment, or to identify physical objects in the user's environment.

[0429] Continuing with reference to FIG. 3, a pair of scanning laser-shaped wavefront (e.g., for depth) light projection modules along with display mirrors and optics are shown configured to project light 338 into the eyes 302, 304. The depicted diagram also shows two miniature infrared cameras 324 paired with infrared light sources 326 (such as light-emitting diodes (LEDs)) configured to track the user's eyes 302, 304 and support rendering and user input. The cameras 324 may be part of the inward-facing imaging system 462 shown in FIG. 4. The wearable system 200 may further feature a sensor assembly 339, which may include X-, Y-, and Z-axis accelerometer capabilities, a magnetic compass, and X-, Y-, and Z-axis gyroscope capabilities, and may preferably provide data at a relatively high frequency, such as 200 Hz. The sensor assembly 339 may be part of an IMU, as described with reference to FIG. 2A. The depicted system 200 may also include a head pose processor 336, such as an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or ARM processor (Advanced Reduced Instruction Set Machine), which may be configured to calculate real-time or near real-time user head pose from the wide FOV image information output from the capture device 316. The head pose processor 336 may be a hardware processor and may be implemented as part of the local processing and data module 260 shown in FIG. 2A.

[0430] The wearable system may also include one or more depth sensors 234. The depth sensors 234 may be configured to measure the distance between objects in the environment and the wearable device. The depth sensors 234 may include a laser scanner (e.g., LIDAR), an ultrasonic depth sensor, or a depth-sensing camera. In some implementations where the camera 316 has depth-sensing capabilities, the camera 316 may also be considered a depth sensor 234.

[0431] Also shown is a processor 332 configured to perform digital or analog processing and derive attitude from gyroscope, compass, or accelerometer data from sensor assembly 339. Processor 332 may be part of local processing and data module 260, shown in FIG. 2. Wearable system 200 may also include a positioning system, such as, for example, a GPS 337 (Global Positioning System), as shown in FIG. 3, to assist in attitude and positioning analysis. In addition, the GPS may further provide remotely based (e.g., cloud-based) information about the user's environment. This information may be used to recognize objects or information within the user's environment.

[0432] The wearable system may combine data obtained by the GPS 337 and a remote computing system (e.g., the remote processing module 270, another user's ARD, etc.), which can provide more information about the user's environment. As one example, the wearable system can determine the user's location based on GPS data and retrieve a world map (e.g., by communicating with the remote processing module 270) that includes virtual objects associated with the user's location. As another example, the wearable system 200 can monitor the environment using the world camera 316 (which may be part of the outward-facing imaging system 464 shown in FIG. 4). Based on the images obtained by the world camera 316, the wearable system 200 can detect objects in the environment (e.g., by using one or more object recognizers 708 shown in FIG. 7). The wearable system can further interpret characters using data obtained by the GPS 337.

[0433] The wearable system 200 may also include a rendering engine 334, which can be configured to provide local rendering information to the user for the user's view of the world and facilitate the operation of the scanner and imaging into the user's eye. The rendering engine 334 may be implemented by a hardware processor (e.g., a central processing unit or a graphics processing unit, etc.). In some embodiments, the rendering engine is part of the local processing and data module 260. The rendering engine 334 can be communicatively coupled to other components of the wearable system 200 (e.g., via wired or wireless links). For example, the rendering engine 334 can be coupled to the eye camera 324 via communication link 274 and to the projection subsystem 318 (which can project light into the user's eyes 302, 304 via a scanning laser array in a manner similar to a retinal scanning display) via communication link 272. The rendering engine 334 can also communicate with other processing units, such as the sensor pose processor 332 and the image pose processor 336, via links 276 and 294, respectively.

[0434] A camera 324 (e.g., a small infrared camera) may be utilized to track eye pose and support rendering and user input. Some example eye poses may include where the user is looking or the depth of focus (which may be estimated using eye convergence and divergence). A GPS 337, gyroscope, compass, and accelerometer 339 may be utilized to provide coarse or fast pose estimation. One or more of the cameras 316 may obtain images and poses, which, along with data from associated cloud computing resources, may be utilized to map the local environment and share the user's view with others.

[0435] The example components depicted in FIG. 3 are for illustrative purposes only. Multiple sensors and other functional modules are shown together for ease of illustration and description. Some embodiments may include only one or a subset of these sensors or modules. Furthermore, the locations of these components are not limited to the locations depicted in FIG. 3. Some components may be mounted or stored within other components, such as belt-mounted, handheld, or helmet-mounted components. As an example, the image pose processor 336, the sensor pose processor 332, and the rendering engine 334 may be located within a beltpack and configured to communicate with other components of the wearable system via wireless communications, such as ultra-wideband, Wi-Fi, Bluetooth, or via wired communications. The depicted housing 230 is preferably head-mountable and wearable by a user. However, some components of the wearable system 200 may be worn on other parts of the user's body. For example, the speaker 240 may be inserted into the user's ear to provide sound to the user.

[0436] With respect to projecting light 338 into the user's eyes 302, 304, in some embodiments, the camera 324 may be utilized to measure where the center of the user's eyes geometrically converges, which generally coincides with the position of the eye's focal point or "depth of focus." The three-dimensional surface of all points at which the eyes converge may be referred to as the "monocular locus." The focal distance may have a finite number of depths or may vary infinitely. Light projected from the convergence distance appears focused on the subject's eyes 302, 304, while light in front of or behind the convergence distance is blurred. Examples of wearable devices and other display systems of the present disclosure are also described in U.S. Patent Publication No. 2016 / 0270656, which is incorporated herein by reference in its entirety.

[0437] The human visual system is complex, making it difficult to provide a realistic perception of depth. A viewer of an object may perceive the object as three-dimensional due to a combination of vergence-divergence shifts and accommodation. Vergence-divergence shifts of the two eyes relative to one another (e.g., pupil rotation, such that the pupils move toward or away from one another, converging the eyes' lines of sight and fixating on an object) are closely linked to the focusing of the eye's lenses (or "accommodation"). Under normal conditions, a change in the focus of the eye's lenses or accommodation of the eye to change focus from one object to another at a different distance will automatically produce a matched change in vergence-divergence at the same distance, a relationship known as the "accommodation-vergence-divergence reflex." Similarly, a change in vergence-divergence will induce a matched change in accommodation under normal conditions. Display systems that provide better matching between accommodation and vergence-divergence may produce more realistic and pleasant simulations of three-dimensional images.

[0438] Furthermore, spatially coherent light with a beam diameter of less than approximately 0.7 millimeters can be correctly resolved by the human eye regardless of where the eye is focused. Therefore, to create the proper illusion of depth of focus, the eye's convergence and divergence movements may be tracked using the camera 324, and the rendering engine 334 and projection subsystem 318 may be utilized to render all objects on or near the single visual path in focus and all other objects variably out of focus (e.g., using intentional blur). Preferably, the system 220 renders to the user at a frame rate of approximately 60 frames per second or greater. As described above, the camera 324 may preferably be utilized for eye tracking, and software may be configured to capture not only convergence and divergence geometry but also focus location cues to serve as user input. Preferably, such a display system is configured with brightness and contrast suitable for daytime or nighttime use.

[0439] In some embodiments, the display system preferably has a latency of less than about 20 milliseconds for visual object alignment, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 arc minute, which, without being limited by theory, is believed to be approximately the limit of the human eye. The display system 220 may be integrated with a localization system, which may involve a GPS element, optical tracking, a compass, an accelerometer, or other data sources to assist in position and attitude determination. The localization information may be utilized to facilitate accurate rendering within the user's view of the relevant world (e.g., such information would help the glasses understand their location relative to the real world).

[0440] In some embodiments, the wearable system 200 is configured to display one or more virtual images based on the accommodation of the user's eyes. Unlike traditional 3D display approaches that force the user to focus where the image is projected, in some embodiments, the wearable system is configured to automatically vary the focus of the projected virtual content, allowing for a more comfortable viewing of one or more images presented to the user. For example, if the user's eyes have a current focus of 1 m, the image may be projected to match the user's focus. If the user shifts focus to 3 m, the image will be projected to match the new focus. Thus, rather than forcing a predetermined focus on the user, the wearable system 200 of some embodiments allows the user's eyes to function in a more natural manner.

[0441] Such a wearable system 200 may eliminate or reduce the incidence of eye strain, headaches, and other physiological symptoms typically observed with virtual reality devices. To achieve this, various embodiments of the wearable system 200 are configured to project virtual images at variable focal lengths through one or more variable focus elements (VFEs). In one or more embodiments, 3D perception may be achieved through a multi-planar focus system that projects images onto a fixed focal plane away from the user. Other embodiments employ a variable planar focus, where the focal plane is moved back and forth in the z-direction to match the user's current state of focus.

[0442] In both multi-plane and variable-plane focus systems, the wearable system 200 may employ eye tracking to determine the convergence and divergence of the user's eyes, determine the user's current focus, and project the virtual image at the determined focus. In other embodiments, the wearable system 200 includes a light modulator that variably projects a variably focused light beam in a raster pattern across the retina through a fiber scanner or other light-generating source. Thus, the wearable system 200's display's ability to project images at variable focal lengths not only facilitates accommodation for the user to view objects in 3D, but may also be used to compensate for the user's ocular abnormalities, as further described in U.S. Patent Publication No. 2016 / 0270656 (incorporated herein by reference in its entirety). In some other embodiments, a spatial light modulator may project an image to the user through various optical components. For example, as further described below, the spatial light modulator may project the image onto one or more waveguides, which then transmit the image to the user.

[0443] Waveguide Stack Assembly FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. Wearable system 400 includes a stack of waveguides or stacked waveguide assembly 480 that can be utilized to provide three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, wearable system 400 may correspond to wearable system 200 of FIG. 2, and FIG. 4 schematically illustrates several portions of wearable system 200 in more detail. For example, in some embodiments, waveguide assembly 480 may be integrated into display 220 of FIG. 2.

[0444] 4, the waveguide assembly 480 may also include multiple features 458, 456, 454, 452 between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers or structures to form air gaps).

[0445] Waveguides 432b, 434b, 436b, 438b, 440b or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye using various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 may be utilized to inject image information into waveguides 440b, 438b, 436b, 434b, 432b, respectively, which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits the output surfaces of image injection devices 420, 422, 424, 426, 428 and is injected into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be injected into each waveguide, outputting an entire field of cloned collimated beams directed toward eye 410 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0446] In some embodiments, image input devices 420, 422, 424, 426, 428 are discrete displays that generate image information for input into each corresponding waveguide 440b, 438b, 436b, 434b, 432b, respectively. In some other embodiments, image input devices 420, 422, 424, 426, 428 are outputs of a single multiplexed display that may, for example, send image information to each of image input devices 420, 422, 424, 426, 428 via one or more optical conduits (such as fiber optic cables).

[0447] A controller 460 controls the operation of stacked waveguide assembly 480 and image injection devices 420, 422, 424, 426, 428. Controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provision of image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, controller 460 may be a single integrated device or a distributed system connected by a wired or wireless communication channel. Controller 460 may, in some embodiments, be part of processing module 260 or 270 (shown in FIG. 2).

[0448] Waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between the major top and bottom surfaces. In the illustrated configuration, waveguides 440b, 438b, 436b, 434b, 432b may each include light extraction optical elements 440a, 438a, 436a, 434a, 432a configured to extract light from the waveguides by redirecting the light, causing it to propagate within each individual waveguide, and outputting image information from the waveguides to the eye 410. The extracted light may also be referred to as out-coupled light, and the light extraction optical element may also be referred to as out-coupling optical element. The extracted light beam is output by the waveguide where the light propagating within the waveguide strikes the light redirecting element. The light extraction optical element (440a, 438a, 436a, 434a, 432a) may be, for example, a reflective or diffractive optical feature. While shown disposed on the bottom major surfaces of the waveguides 440b, 438b, 436b, 434b, 432b for ease of explanation and clarity of drawing, in some embodiments, the light extraction optical element 440a, 438a, 436a, 434a, 432a may be disposed on the top or bottom major surfaces, or directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed in a layer of material attached to a transparent substrate and forming the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be monolithic pieces of material, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within the material pieces.

[0449] Continuing with reference to FIG. 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is launched into such waveguide 432b. The collimated light may represent an optical infinity focal plane. The next upper waveguide 434b may be configured to send collimated light that passes through a first lens 452 (e.g., a negative lens) before reaching the eye 410. The first lens 452 may be configured to create a slight convex wavefront curvature so that the eye / brain interprets light emerging from the next upper waveguide 434b as emerging from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third upper waveguide 436b passes its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to produce another, increasing amount of wavefront curvature such that the eye / brain interprets the light emerging from the third waveguide 436b as originating from a second focal plane that is closer inward from optical infinity towards the person than was the light from the next upper waveguide 434b.

[0450] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the highest waveguide 440b in the stack used to send its output through all of the lenses between it and the eye for a collective focal power representing the focal plane closest to the person. To compensate for the stack of lenses 458, 456, 454, 452 when viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be placed on top of the stack to compensate for the collective power of the lower lens stacks 458, 456, 454, 452. (The compensatory lens layer 430 and stacked waveguide assembly 480 may be configured collectively so that light originating from the world 470 is transmitted to the eye 410 with substantially the same level of divergence (or collimation) as the light had when originally received by the stacked waveguide assembly 480.) Such a configuration provides as many perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electro-active). In some alternative embodiments, one or both may be dynamic using electro-active features.

[0451] Continuing with reference to FIG. 4 , light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to redirect light from their respective waveguides and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with that waveguide. As a result, waveguides with different associated depth planes may have differently configured light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, light extraction optical elements 440a, 438a, 436a, 434a, 432a may be solid or surface features that can be configured to output light at specific angles. For example, light extraction optical elements 440a, 438a, 436a, 434a, 432a may be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015, which is incorporated herein by reference in its entirety.

[0452] In some embodiments, light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency so that only a portion of the light in the beam is deflected toward the eye 410 at each intersection of the DOE, while the remainder continues traveling through the waveguide via total internal reflection. The light carrying the image information is thus split into several related output beams that exit the waveguide at multiple locations, which can result in a very uniform pattern of output emission toward the eye 304 for this particular collimated beam bouncing within the waveguide.

[0453] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal in which microdroplets comprise a diffractive pattern in a host medium, and the refractive index of the microdroplets may be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets may be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0454] In some embodiments, the number and distribution of depth planes or depths of field may be dynamically varied based on the size or orientation of the viewer's pupil. The depth of field may vary inversely with the viewer's pupil size. As a result, as the size of the viewer's pupil decreases, the depth of field increases so that a plane that is indistinguishable because its location exceeds the eye's depth of focus may become distinguishable and appear more focused with a corresponding decrease in pupil size and an increase in depth of field. Similarly, the number of spaced depth planes used to present different images to the viewer may be reduced with a decreased pupil size. For example, a viewer may not be able to clearly perceive details in both a first depth plane and a second depth plane at one pupil size without adjusting their eye's accommodation from one depth plane to the other. However, these two depth planes may be sufficient to simultaneously focus on the user at another pupil size without changing accommodation.

[0455] In some embodiments, the display system may vary the number of waveguides receiving image information based on a determination of pupil size or orientation or in response to receiving an electrical signal indicating a particular pupil size or orientation. For example, if a user's eye is unable to distinguish between two depth planes associated with two waveguides, controller 460 (which may be an embodiment of local processing and data module 260) may be configured or programmed to stop providing image information to one of those waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing system responsiveness. In embodiments in which the DOE for a waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0456] In some embodiments, it may be desirable to have the output beam satisfy the condition of having a diameter less than the diameter of the viewer's eye. However, meeting this condition may be difficult in light of the variability in the size of the viewer's pupil. In some embodiments, this condition is met over a wide range of pupil sizes by varying the size of the output beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the output beam may also decrease. In some embodiments, the output beam size may be varied using a variable aperture.

[0457] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the world camera's field of view (FOV), and the imaging system 464 is sometimes also referred to as the FOV camera. The world camera's FOV may or may not be the same as the viewer 210's FOV, which encompasses the portion of the world 470 that the viewer 210 perceives at a given time. For example, in some situations, the world camera's FOV may be larger than the viewer 210's field of view of the wearable system 400. The entire area available for viewing or imaging by the viewer may be referred to as the field of view (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400, since the wearer may move their body, head, or eyes and perceive virtually any direction in space. In other contexts, the wearer's movement may be more constrained, and the wearer's FOR may correspondingly subtend a smaller solid angle. Images obtained from the outward-facing imaging system 464 can be used to track gestures (e.g., hand or finger gestures) made by the user, detect objects in the world 470 in front of the user, etc.

[0458] The wearable system 400 includes an audio sensor 232, e.g., a microphone, that can capture ambient sounds. As described above, in some embodiments, one or more other audio sensors can be positioned to provide stereo sound reception useful in determining the location of a speech source. As another example, the audio sensor 232 can include a directional microphone, which can also provide such useful directional information regarding where an audio source is located. The wearable system 400 can use information from both the outward-facing imaging system 464 and the audio sensor 230 in locating a speech source or to determine the active speaker at a particular moment, etc. For example, the wearable system 400 can use voice recognition, alone or in combination with a reflected image of the speaker (e.g., as seen in a mirror), to determine the identity of the speaker. As another example, the wearable system 400 can determine the location of a speaker in the environment based on sound obtained from a directional microphone. The wearable system 400 can use speech recognition algorithms to analyze the sounds coming from the speaker's location, determine the content of the speech, and use voice recognition techniques to determine the speaker's identity (e.g., name or other demographic information).

[0459] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes user movements, such as eye and facial movements. The inward-facing imaging system 466 may be used to capture images of the eyes 410 and determine the size and / or orientation of the pupils of the eyes 304. The inward-facing imaging system 466 may be used to obtain images for use in determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera may be utilized for each eye independently to separately determine the pupil size or eye pose of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter or orientation of only a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. Images obtained by inward-facing imaging system 466 may be analyzed to determine the user's eye posture or mood, which may be used by wearable system 400 to determine audio or visual content to be presented to the user. Wearable system 400 may also determine head pose (e.g., head position or head orientation) using sensors such as an IMU, accelerometer, gyroscope, etc.

[0460] The wearable system 400 may include a user input device 466 through which a user may input commands into the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, touchscreen, joystick, multi-degree-of-freedom (DOF) controller, capacitive sensing device, game controller, keyboard, mouse, directional pad (D-pad), wand, tactile device, totem (e.g., functioning as a virtual user input device), etc. A multi-DOF controller may sense user input in possible translation (e.g., left / right, forward / backward, or up / down) or rotation (e.g., yaw, pitch, or roll) of some or all of the controller. A multi-DOF controller that supports translation may be referred to as 3DOF, while a multi-DOF controller that supports translation and rotation may be referred to as 6DOF. In some cases, a user may use a finger (e.g., a thumb) to press or swipe across a touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 via wired or wireless communication.

[0461] Other Components of a Wearable System In many implementations, the wearable system may include other components in addition to or as an alternative to the components of the wearable system described above. The wearable system may include, for example, one or more tactile devices or components. The tactile device or component may be operable to provide a tactile sensation to the user. For example, the tactile device or component may provide a tactile sensation of pressure or texture upon touching virtual content (e.g., a virtual object, virtual tool, other virtual structure). The tactile sensation may replicate the sensation of a physical object represented by the virtual object, or may replicate the sensation of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some implementations, the tactile device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the tactile device or component may be held by the user.

[0462] A wearable system may include, for example, one or more physical objects that can be manipulated by a user to enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, a piece of metal or plastic, a wall, a table surface, etc. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, the totem may simply provide a physical surface, and the wearable system may render a user interface to appear to the user on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad to appear on the surface of a thin rectangular plate of aluminum that serves as the totem. The rectangular plate itself does not have any physical keys, trackpads, or sensors. However, the wearable system may detect user manipulation or interaction or touch with the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. User input device 466 (shown in FIG. 4) may be an embodiment of a totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. A user may use the totem alone or in combination with posture to interact with the wearable system or other users.

[0463] Examples of tactile devices and totems usable with the wearable devices, HMDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety.

[0464] Example of eye image FIG. 5 illustrates an image of an eye 500 with an eyelid 504, a sclera 508 ("white of the eye"), an iris 512, and a pupil 516. Curve 516a indicates the pupillary boundary between the pupil 516 and the iris 512, and curve 512a indicates the peripheral boundary between the iris 512 and the sclera 508. The eyelid 504 includes an upper eyelid 504a and a lower eyelid 504b. The eye 500 is illustrated in a natural resting position (e.g., oriented such that both the user's face and gaze would be directed toward a far-distance object directly in front of the user). The natural resting position of the eye 500 may be represented by a natural resting direction 520, which is a direction perpendicular to the surface of the eye 500 when in a natural resting position (e.g., immediately out-of-plane for the eye 500 shown in FIG. 5) and, in this example, centered within the pupil 516.

[0465] As the eye 500 moves to look toward different objects, the eye pose will change relative to the natural resting direction 520. The current eye pose can be determined with reference to an eye pose direction 524, which is a direction orthogonal to the surface of the eye (and centered within the pupil 516), but oriented toward the object at which the eye is currently pointed. With reference to the exemplary coordinate system shown in FIG. 5A , the pose of the eye 500 can be represented as two angular parameters indicating the azimuth and zenith deflections of the eye's eye pose direction 524, both relative to the eye's natural resting direction 520. For illustrative purposes, these angular parameters can be represented as θ (the azimuth deflection, determined from the origin azimuth angle) and φ (the zenith deflection, sometimes also referred to as the polar deflection). In some implementations, the angular roll of the eye about the eye pose direction 524 can be included in the determination of eye pose, and the angular roll can be included in the following analysis. In other implementations, other techniques for determining eye pose can be used, for example, pitch, yaw, and optionally roll systems.

[0466] The eye images can be obtained from the video using any suitable process, for example, using a video processing algorithm that can extract images from one or more sequential frames. The eye pose can be determined from the eye images using various eye tracking techniques. For example, the eye pose can be determined by considering the lens effect of the cornea on the provided light source. Any suitable eye tracking technique can be used to determine the eye pose in the eyelid shape estimation techniques described herein.

[0467] Example of an eye tracking system FIG. 6 illustrates a schematic diagram of a wearable system 600 including an eye tracking system. The wearable system 600, in at least some embodiments, includes components located in a head-mounted unit 602 and components located in a non-head-mounted unit 604. The non-head-mounted unit 604 may be, by way of example, a belt-mounted component, a handheld component, a component in a backpack, a remote component, etc. Incorporating some of the components of the wearable system 600 into the non-head-mounted unit 604 may help reduce the size, weight, complexity, and cost of the head-mounted unit 602. In some implementations, some or all of the functionality described as being performed by one or more components of the head-mounted unit 602 and / or the non-head-mounted unit 604 may be provided using one or more components included anywhere within the wearable system 600. For example, some or all of the functionality described below in connection with CPU 612 of head-mounted unit 602 may be provided using CPU 616 of non-head-mounted unit 604, or vice versa. In some embodiments, some or all of such functionality may be provided using peripheral devices of wearable system 600. Furthermore, in some implementations, some or all of such functionality may be provided using one or more cloud computing devices or other remotely located computing devices, in a manner similar to that described above with reference to FIG. 2.

[0468] As shown in FIG. 6 , the wearable system 600 may include an eye tracking system including a camera 324 that captures images of the user's eyes 610. If desired, the eye tracking system may also include light sources 326 a and 326 b (such as light-emitting diodes (LEDs)). The light sources 326 a and 326 b may generate a flash of light (e.g., a reflection from the user's eye that appears in an image of the eye captured by the camera 324). The position of the light sources 326 a and 326 b relative to the camera 324 may be known, so that the position of the flash of light in the image captured by the camera 324 may be used in tracking the user's eyes (as will be discussed in more detail below in connection with FIGS. 7-11 ). In at least one embodiment, there may be one light source 326 and one camera 324 associated with one of the user's eyes 610. In another embodiment, there may be one light source 326 and one camera 324 associated with each of the user's eyes 610. In still other embodiments, there may be one or more cameras 324 and one or more light sources 326 associated with one or each of the user's eyes 610. As a specific example, there may be two light sources 326a and 326b and one or more cameras 324 associated with each of the user's eyes 610. As another example, there may be three or more light sources, such as light sources 326a and 326b, and one or more cameras 324 associated with each of the user's eyes 610. The eye tracking module 614 may receive images from the eye tracking cameras 324 and may analyze the images to extract various information. As examples, the eye tracking module 614 may detect the user's eye posture, the three-dimensional position of the user's eyes relative to the eye tracking camera 324 (and head-mounted unit 602), the direction in which one or both of the user's eyes 610 are focused, the user's convergence and divergence depth (e.g., the depth from the user at which the user is focused), the position of the user's pupils, the position of the user's cornea and corneal sphere, the center of rotation of each of the user's eyes, and the center of gaze of each of the user's eyes. The eye tracking module 614 may extract such information using techniques described below in connection with Figures 7-11.As shown in FIG. 6, the eye tracking module 614 may be a software module implemented using the CPU 612 in the head-mounted unit 602.

[0469] Data from the eye tracking module 614 may be provided to other components within the wearable system. As an example, such data may be transmitted to components within the non-head-mounted unit 604, such as the CPU 616, including software modules for a light field rendering controller 618 and an alignment observer 620.

[0470] The rendering controller 618 may use information from the eye tracking module 614 to adjust the images displayed to the user via the rendering engine 622 (e.g., the rendering engine, which may be a software module within the GPU 620 and may provide images to the display 220). As an example, the rendering controller 618 may adjust the images displayed to the user based on the user's center of rotation or center of viewpoint. In particular, the rendering controller 618 may use information about the user's center of viewpoint to simulate a rendering camera (e.g., to simulate the collection of images from the user's viewpoint) and adjust the images displayed to the user based on the simulated rendering camera.

[0471] A "rendering camera," sometimes referred to as a "pinhole perspective camera" (or simply, a "perspective camera") or a "virtual pinhole camera" (or simply, a "virtual camera"), is a simulated camera for use in rendering virtual image content, possibly from a database of objects in a virtual world. The objects may have a location and orientation relative to a user or wearer, and possibly relative to real objects in the environment surrounding the user or wearer. In other words, the rendering camera may represent a viewpoint in the rendering space from which the user or wearer should view the 3D virtual content (e.g., virtual objects) of the rendering space. The rendering camera may render a virtual image based on a database of virtual objects to be presented to the eye, managed by a rendering engine. The virtual image may be rendered as if taken from the user's or wearer's perspective. For example, a virtual image may be rendered as if it were captured by a pinhole camera (corresponding to a "rendering camera") having a specific set of intrinsic parameters (e.g., focal length, camera pixel size, principal point coordinates, distortion / distortion parameters, etc.) and a specific set of extrinsic parameters (e.g., translation and rotation components relative to the virtual world). The virtual image is captured from the viewpoint of such a camera having the rendering camera's position and orientation (e.g., the rendering camera's extrinsic parameters). It follows that the system may define and / or adjust the intrinsic and extrinsic rendering camera parameters. For example, the system may define a particular set of extrinsic rendering camera parameters such that the virtual image is rendered as if it were captured from the viewpoint of a camera having a specific location relative to the user's or wearer's eyes to provide an image that appears as if it were from the user's or wearer's perspective. The system may later dynamically adjust the extrinsic rendering camera parameters on the fly to maintain alignment with the specific location. Similarly, intrinsic rendering camera parameters may also be defined and dynamically adjusted over time.In some implementations, the image is rendered as if it were captured from the viewpoint of a camera having an aperture (e.g., a pinhole) at a specific location (such as the center of viewpoint or center of rotation or other location) relative to the user's or wearer's eyes.

[0472] In some embodiments, the system may create or dynamically reposition and / or reorient one rendering camera for the user's left eye and another rendering camera for the user's right eye as the user's eyes are physically separated from one another and therefore consistently positioned in different locations. In at least some implementations, virtual content rendered from the perspective of a rendering camera associated with the viewer's left eye may be presented to the user through a left eyepiece of a head-mounted display (e.g., head-mounted unit 602), and virtual content rendered from the perspective of a rendering camera associated with the user's right eye may be presented to the user through a right eyepiece of such head-mounted display. Further details discussing the creation, adjustment, and use of rendering cameras in the rendering process are provided in U.S. Patent Application No. 15 / 274,823, entitled "METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN 3D RECONSTRUCTION," which is expressly incorporated herein by reference in its entirety for all purposes.

[0473] In some embodiments, one or more modules (or components) of system 600 (e.g., light field rendering controller 618, rendering engine 620, etc.) may determine the position and orientation of a rendering camera within a rendering space based on the position and orientation of the user's head and eyes (e.g., as determined based on head pose and eye tracking data, respectively). That is, system 600 may, in effect, map the user's head and eye position and orientation to a specific location and angular position within the 3D virtual environment, install and orient the rendering camera to a specific location and angular position within the 3D virtual environment, and render virtual content for the user as it may be captured by the rendering camera. Further details discussing the real-world / virtual-world mapping process are provided in U.S. Patent Application No. 15 / 296,869, entitled "SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE," which is expressly incorporated herein by reference in its entirety for all purposes. As an example, the rendering controller 618 may adjust the depth at which an image is displayed by selecting a depth plane (or multiple depth planes) to be utilized at any given time to display the image. In some implementations, such depth plane switching may be performed through adjustment of one or more intrinsic rendering camera parameters. For example, the light field rendering controller 618 may adjust the focal length of the rendering camera when performing depth plane switching or adjustment. As described in further detail below, depth planes may be switched based on the user's determined convergence-divergence movement or fixation depth.

[0474] The alignment observer 620 may use information from the eye tracking module 614 to identify whether the head-mounted unit 602 is properly positioned on the user's head. As an example, the eye tracking module 614 may provide eye location information, such as the location of the center of rotation of the user's eyes, which indicates the three-dimensional position of the user's eyes relative to the camera 324, and the head-mounted unit 602 and eye tracking module 614 may use the location information to determine whether the display 220 is properly aligned within the user's field of view or whether the head-mounted unit 602 (or headset) has slipped or is otherwise misaligned with the user's eyes. As examples, alignment observer 620 may be able to determine whether head-mounted unit 602 has slipped off the bridge of the user's nose, thus moving display 220 away from and downwardly away from the user's eyes (which may be undesirable), whether head-mounted unit 602 has moved above the bridge of the user's nose, thus moving display 220 closer to and upwardly away from the user's eyes, whether head-mounted unit 602 has been shifted left or right relative to the bridge of the user's nose, whether head-mounted unit 602 has been lifted above the bridge of the user's nose, or whether head-mounted unit 602 has been moved away from a desired position or range of positions in these or other ways. In general, alignment observer 620 may be able to determine whether head-mounted unit 602, and display 220 in particular, are properly positioned directly in front of the user's eyes. In other words, alignment observer 620 may determine whether the left display in display system 220 is properly aligned with the user's left eye, and whether the right display in display system 220 is properly aligned with the user's right eye. Alignment observer 620 may determine whether head-mounted unit 602 is properly positioned by determining whether head-mounted unit 602 is positioned and oriented within a desired range of position and / or orientation relative to the user's eyes.

[0475] In at least some embodiments, alignment observer 620 may generate user feedback in the form of alerts, messages, or other content. Such feedback may be provided to the user to inform the user of any misalignment of head-mounted unit 602, along with optional feedback on how to correct the misalignment (such as suggestions to adjust head-mounted unit 602 in a particular manner).

[0476] Exemplary alignment observation and feedback techniques that may be utilized by alignment observer 620 are described in U.S. patent application Ser. No. 15 / 717,747, filed Sep. 27, 2017 (Attorney Docket No. MLEAP.052A2), and U.S. provisional patent application Ser. No. 62 / 644,321, filed Mar. 16, 2018 (Attorney Docket No. MLEAP.195PR), both of which are incorporated by reference in their entireties herein.

[0477] Example of an eye tracking module A detailed block diagram of an exemplary eye tracking module 614 is shown in Figure 7A. As shown in Figure 7A, the eye tracking module 614 may include a variety of different sub-modules, may provide a variety of different outputs, and may utilize a variety of available data in tracking the user's eyes. As an example, the eye tracking module 614 may utilize available data, including extrinsic and intrinsic properties of eye tracking, such as the geometry of the eye tracking camera 324 relative to the light source 326 and head-mounted unit 602, assumed eye dimensions 704, such as a typical distance of approximately 4.7 mm between the center of curvature of the user's cornea and the average center of rotation of the user's eye or a typical distance between the user's center of rotation and the center of gaze, and per-user calibration data 706, such as the interpupillary distance of a particular user. Additional examples of extrinsic properties, intrinsic properties, and other information that may be employed by the eye tracking module 614 are described in U.S. Patent Application No. 15 / 497,726, filed April 26, 2017 (Attorney Docket No. MLEAP.023A7), which is incorporated herein by reference in its entirety.

[0478] Image preprocessing module 710 may receive images from an eye camera, such as eye camera 324, and may perform one or more preprocessing (e.g., adjustment) operations on the received images. As examples, image preprocessing module 710 may apply Gaussian blur to the images, downsample the images to a lower resolution, apply an unsharp mask, apply an edge sharpening algorithm, or apply other suitable filters to aid in the subsequent detection, location, and labeling of phosphenes, pupils, or other features in images from eye camera 324. Image preprocessing module 710 may apply a low-pass filter or a morphological filter, such as an open filter, which may remove high-frequency noise from pupil boundary 516a (see FIG. 5), thereby removing noise that may interfere with pupil and phosphene determination. Image preprocessing module 710 may output the preprocessed images to pupil identification module 712 and phosphene detection and labeling module 714.

[0479] The pupil identification module 712 may receive preprocessed images from the image preprocessing module 710 and may identify regions of those images that contain the user's pupil. The pupil identification module 712, in some embodiments, may determine the coordinates of the location of the user's pupil within the eye tracking images from the camera 324, i.e., the coordinates of the center or centroid. In at least some embodiments, the pupil identification module 712 may identify contours within the eye tracking images (e.g., the contours of the pupil-iris boundary), identify contour moments (e.g., the center of mass), apply starburst pupil detection and / or Canny edge detection algorithms, filter out outliers based on intensity values, identify sub-pixel boundary points, correct for eye camera distortion (e.g., distortion in images captured by the eye camera 324), apply a random sample consensus (RANSAC) iterative algorithm, fit ellipses to boundaries within the eye tracking images, apply a tracking filter to the images, and identify sub-pixel image coordinates of the user's pupil centroid. Pupil identification module 712 may output pupil identification data, which may indicate regions of preprocessed image module 712 that have been identified as indicative of the user's pupil, to flash detection and labeling module 714. Pupil identification module 712 may provide 2D coordinates of the user's pupil in each eye tracking image (e.g., 2D coordinates of the centroid of the user's pupil) to flash detection module 714. In at least some embodiments, pupil identification module 712 may also provide the same type of pupil identification data to coordinate system normalization module 718.

[0480] Pupil detection techniques that may be utilized by pupil identification module 712 are described in U.S. Patent Publication No. 2017 / 0053165, published February 23, 2017, and U.S. Patent Publication No. 2017 / 0053166, published February 23, 2017 (each of which is incorporated by reference in its entirety herein).

[0481] The flash detection and labeling module 714 may receive the preprocessed image from module 710 and the pupil identification data from module 712. The flash detection module 714 may use this data to detect and / or identify flashes (i.e., reflections of light from the light source 326 off the user's eye) in areas of the preprocessed image that represent the user's pupil. As an example, the flash detection module 714 may search for bright areas, sometimes referred to herein as "blobs" or local intensity maxima, in the eye tracking image that are in the vicinity of the user's pupil. In at least some embodiments, the flash detection module 714 may rescale (e.g., expand) the pupil ellipse to include additional flashes. The flash detection module 714 may filter flashes by size and / or intensity. The flash detection module 714 may also determine the 2D location of each flash within the eye tracking image. In at least some embodiments, the flash detection module 714 may determine the 2D position of the flash relative to the user's pupil, which may also be referred to as the pupil-flashing vector. The flash detection and labeling module 714 may label the flashes and output a preprocessed image with the labeled flashes to the 3D corneal center estimation module 716. The flash detection and labeling module 714 may also pass on data such as the preprocessed image from module 710 and the pupil identification data from module 712. In some implementations, the flash detection and labeling module 714 may determine the light source (e.g., among multiple light sources in the system, including infrared light sources 326a and 326b) that produced each identified flash. In these embodiments, the flash detection and labeling module 714 may label the flashes with information identifying the associated light source and output a preprocessed image with the labeled flashes to the 3D corneal center estimation module 716.

[0482] Pupil and phosphene detection, as performed by modules such as modules 712 and 714, can use any suitable technique. As an example, edge detection can be applied to the eye image to identify phosphenes and pupils. Edge detection can be applied by various edge detectors, edge detection algorithms, or filters. For example, a Canny edge detector can be applied to the image to detect edges, such as lines, in the image. Edges may include points located along the lines that correspond to local maximum derivatives. For example, pupil boundary 516a (see FIG. 5) can be located using a Canny edge detector. Once the location of the pupil is determined, various image processing techniques can be used to detect the “pose” of pupil 116. Determining the eye pose of the eye image may also be referred to as detecting the eye pose of the eye image. Pose may also be referred to as gaze, facing direction, or eye orientation. For example, the pupil may be looking left toward an object, and the pupil pose may be classified as a left-looking pose. Other methods can also be used to detect the location of the pupil or phosphene. For example, concentric rings may be located in an eye image using a Canny edge detector. As another example, an integro-differential operator may be used to find the corneal or limbal boundary of the pupil or iris. For example, a Daugman integro-differential operator, a Hough transform, or other iris segmentation techniques can be used to return a curve that estimates the boundary of the pupil or iris.

[0483] The 3D corneal center estimation module 716 may receive preprocessed images from modules 710, 712, and 714, including detected phosphene data and pupil identification data. The 3D corneal center estimation module 716 may use these data to estimate the 3D position of the user's cornea. In some embodiments, the 3D corneal center estimation module 716 may estimate the 3D position of the center of the eye's corneal curvature or the user's corneal sphere, e.g., the center of an imaginary sphere having a surface portion generally coextensive with the user's cornea. The 3D corneal center estimation module 716 may provide data indicating the estimated 3D coordinates of the corneal sphere and / or the user's cornea to the coordinate system normalization module 718, the optical axis determination module 722, and / or the light field rendering controller 618. Further details of the operation of the 3D corneal center estimation module 716 are provided herein in connection with FIGS. 8A-8E . Techniques for estimating the position of ocular features, such as the cornea or corneal sphere, that may be utilized by the 3D corneal center estimation module 716 and other modules in the wearable system of the present disclosure are discussed in U.S. Patent Application No. 15 / 497,726, filed April 26, 2017 (Attorney Docket No. MLEAP.023A7), which is incorporated herein by reference in its entirety.

[0484] Coordinate system normalization module 718 may optionally be included within eye tracking module 614 (as indicated by its dashed outline). Coordinate system normalization module 718 may receive data indicating the estimated 3D coordinates of the center of the user's cornea (and / or the center of the user's corneal sphere) from 3D corneal center estimation module 716, and may also receive data from other modules. Coordinate system normalization module 718 may normalize the eye camera coordinate system, which may help to compensate for slippage of the wearable device (e.g., slippage of a head-mounted component from its normal resting position on the user's head, which may be identified by alignment observer 620). The coordinate system normalization module 718 may rotate the coordinate system to align the z-axis (e.g., the vergence-divergence depth axis) of the coordinate system with the corneal center (e.g., as indicated by the 3D corneal center estimation module 716) and may translate the camera center (e.g., the origin of the coordinate system) a predetermined distance away from the corneal center, such as 30 mm (e.g., module 718 may zoom in or out on the eye tracking image depending on whether the eye camera 324 is determined to be closer or farther than the predetermined distance). Using this normalization process, the eye tracking module 614 may be able to establish consistent orientations and distances in the eye tracking data relatively independent of variations in the headset positioning on the user's head. The coordinate system normalization module 718 may provide the 3D coordinates of the center of the cornea (and / or corneal sphere), pupil identification data, and preprocessed eye tracking images to the 3D pupil center locator module 720. Further details of the operation of the coordinate system normalization module 718 are provided herein in connection with Figures 9A-9C.

[0485] The 3D pupil center locator module 720 may receive data including the 3D coordinate of the center of the user's cornea (and / or corneal sphere), pupil location data, and preprocessed eye tracking images in a normalized or non-normalized coordinate system. The 3D pupil center locator module 720 may analyze such data to determine the 3D coordinate of the center of the user's pupil in a normalized or non-normalized eye camera coordinate system. The 3D pupil center locator module 720 may determine the location of the user's pupil in three dimensions based on the 2D location of the pupil centroid (as determined by module 712), the 3D location of the corneal center (as determined by module 716), assumed eye dimensions 704 such as the size of a typical user's corneal sphere and the typical distance from the corneal center to the pupil center, and optical properties of the eye such as the refractive index of the cornea (relative to the refractive index of air), or any combination thereof. Further details of the operation of 3D pupil center locator module 720 are provided herein in connection with Figures 9D-9G. Techniques for estimating the position of eye features, such as the pupil, that may be utilized by 3D pupil center locator module 720 and other modules in the wearable system of the present disclosure are discussed in U.S. Patent Application No. 15 / 497,726, filed April 26, 2017 (Attorney Docket No. MLEAP.023A7), which is incorporated herein by reference in its entirety.

[0486] Optical axis determination module 722 may receive data from modules 716 and 720 indicating the 3D coordinates of the user's cornea and the center of the user's pupil. Based on such data, optical axis determination module 722 may identify a vector from the location of the corneal center (e.g., from the center of the corneal sphere) to the center of the user's pupil, which may define the optical axis of the user's eye. Optical axis determination module 722 may provide outputs to modules 724, 728, 730, and 732 that define the user's optical axis, as an example.

[0487] The center of rotation (CoR) estimation module 724 may receive data from module 722 including parameters of the optical axis of the user's eye (e.g., data indicating the orientation of the optical axis in a coordinate system with a known relationship to the head-mounted unit 602). For example, the CoR estimation module 724 may estimate the center of rotation of the user's eye. The center of rotation may indicate a point around which the user's eye rotates as the user's eye rotates left, right, up, and / or down. Even if the eye cannot rotate perfectly around a single point, it is assumed that a single point may be sufficient. In at least some embodiments, the CoR estimation module 724 may estimate the center of rotation of the eye by moving a specific distance along the optical axis (identified by module 722) from the center of the pupil (identified by module 720) or the center of curvature of the cornea (as identified by module 716) toward the retina. This specific distance may be the assumed eye dimension 704. As an example, a particular distance between the corneal center of curvature and the CoR may be approximately 4.7 mm. This distance may be varied for a particular user based on any relevant data, including the user's age, gender, vision prescription, other relevant characteristics, etc. Additional discussion of the 4.7 mm value as an estimate for the distance between the corneal center of curvature and the CoR is provided in the Appendix (Section III), which forms a part of this application.

[0488] In at least some embodiments, the CoR estimation module 724 may refine its estimate of the center of rotation of each of the user's eyes over time. As an example, over time, the user will eventually rotate their eyes (to look elsewhere, closer, or further away, or to the left, right, up, or down at certain times), causing a shift in the optical axis of each of their eyes. The CoR estimation module 724 may then analyze the two (or more) optical axes identified by module 722 and locate the 3D point of intersection of those optical axes. The CoR estimation module 724 may then determine a center of rotation that is at that 3D point of intersection. Such a technique may provide an estimate of the center of rotation with improving accuracy over time.

[0489] Various techniques may be employed to increase the accuracy of the CoR estimation module 724 and the determined CoR positions of the left and right eyes. As an example, the CoR estimation module 724 may estimate the CoR by finding the average point of intersection of the optical axes determined over time for a variety of different eye postures. As an additional example, the module 724 may filter or average the estimated CoR positions over time, calculate a moving average of the estimated CoR positions over time, and / or apply a Kalman filter and known dynamics of the eye and eye tracking system to estimate the CoR positions over time. In some implementations, a least-squares approach may be taken to determine one or more points of intersection of the optical axes. In such implementations, the system may identify locations where, at a given time, the sum of the squared distances to a given set of optical axes is reduced or minimized as points of intersection of the optical axes. As a specific example, module 724 may calculate a weighted average of the determined point of optical axis intersection and the assumed CoR location (e.g., 4.7 mm behind the center of the eye's corneal curvature) so that the determined CoR may slowly shift over time from the assumed CoR location (e.g., 4.7 mm behind the center of the eye's corneal curvature) to a slightly different location within the user's eye as eye tracking data is acquired for the user, thereby allowing for per-user refinement of the CoR location.

[0490] Under ideal conditions, the 3D position of the true CoR of a user's eye relative to the HMD should change by a negligible or minimal amount over time as the user moves their eye (e.g., as the user's eye rotates around its center of rotation). In other words, for a given set of eye movements, the 3D position of the true CoR of a user's eye (relative to the HMD) should hypothetically vary less over time than any other point along the optical axis of the user's eye. Thus, it follows that the farther a point along the optical axis is from the true CoR of the user's eye, the more variation or variance its 3D position will exhibit over time as the user moves their eye. In some embodiments, the CoR estimation module 724 and / or other sub-modules of the eye tracking module 614 may utilize this statistical relationship to improve CoR estimation accuracy. In such an embodiment, the CoR estimation module 724 and / or other sub-modules of the eye tracking module 614 may refine its estimate of the CoR 3D position over time by identifying variations in its CoR estimates that have low variations (e.g., low variance or standard deviation).

[0491] As a first example, in an embodiment in which the CoR estimation module 724 estimates the CoR based on the intersection of multiple different optical axes (each associated with the user looking in a different direction), the CoR estimation module 724 may take advantage of this statistical relationship (the true CoR should have low variance) by introducing a common offset into the orientation of each of the optical axes (e.g., shifting each axis by a uniform amount) and determining whether the offset optical axes intersect each other at an intersection that has low variation, e.g., low variance or standard deviation. This can help correct for small systematic errors in the calculation of the orientation of the optical axes and refine the estimated location of the CoR to be closer to the true CoR.

[0492] As a second example, in embodiments in which the CoR estimation module 724 estimates the CoR by moving a particular distance (e.g., the distance between the corneal center of curvature and the CoR) along the optical axis (or other axis), the system may vary, optimize, adjust, or otherwise adjust the particular distance between the corneal center of curvature and the CoR over time (e.g., for a large group of images of the eye captured at different times) in a manner to reduce or minimize the amount of variation, e.g., variance and / or standard deviation, of the estimated CoR position. For example, if the CoR estimation module 724 initially uses a particular distance value of 4.7 mm (from the corneal center of curvature along the optical axis) to obtain CoR position estimates, but the true CoR of a given user's eye may be located 4.9 mm behind the eye's center of corneal curvature (along the optical axis), the initial set of CoR position estimates obtained by the CoR estimation module 724 may exhibit a relatively high amount of variation, e.g., variance or standard deviation. In response to detecting such a relatively high amount of variation (e.g., variance or standard deviation), the CoR estimation module 724 may search for and identify one or more points along the optical axis having a lower amount of variation (e.g., variance or standard deviation), and may identify the 4.9 mm distance as having the lowest amount of variation (e.g., variance or standard deviation), and may adjust the particular distance value utilized to 4.9 mm accordingly.

[0493] The CoR estimation module 724 may search for alternative CoR estimates with lower variability (e.g., variance and / or standard deviation) in response to detecting that the current CoR estimate has a relatively high amount of variability (e.g., variance or standard deviation), or may search for alternative CoR estimates with lower variability (e.g., variance and / or standard deviation) as appropriate after obtaining an initial CoR estimate. In some embodiments, such optimization / adjustment may occur gradually over time, while in other embodiments, such optimization / adjustment may be performed during an initial user calibration session. In embodiments in which such a procedure is performed during a calibration procedure, the CoR estimation module 724 may not initially agree with / adhere to any assumed particular distance, but rather may collect sets of eye tracking data over time, perform statistical analysis on the sets of eye tracking data, and, based on the statistical analysis, determine a particular distance value that results in a CoR position estimate with the least possible amount of variability (e.g., variance or standard deviation) (e.g., a global minimum).

[0494] Additional discussion of the statistical relationships described above (e.g., true CoR should have a low variance or standard deviation) and the significance of considering corneal refraction in determining pupil position is provided in the Appendix (Chapter III), which forms a part of this application.

[0495] The interpupillary distance (IPD) estimation module 726 may receive data from the CoR estimation module 724 indicating estimated 3D positions of the centers of rotation of the user's left and right eyes. The IPD estimation module 726 may then estimate the user's IPD by measuring the 3D distance between the centers of rotation of the user's left and right eyes. Generally, the distance between the estimated CoR of the user's left eye and the estimated CoR of the user's right eye may be approximately equal to the distance between the centers of the user's pupils when the user is looking at optical infinity (e.g., the optical axes of the user's eyes are approximately parallel to each other), which is a typical definition of the interpupillary distance (IPD). The user's IPD may be used by various components and modules within the wearable system. As an example, the user's IPD may be provided to the alignment observer 620 and used in assessing the degree to which the wearable device is aligned with the user's eyes (e.g., whether the left and right display lenses are properly spaced according to the user's IPD). As another example, the user's IPD may be provided to the convergence-divergence depth estimation module 728 and used in determining the user's convergence-divergence depth. The module 726 may employ various techniques to increase the accuracy of the estimated IPD, such as those discussed in connection with the CoR estimation module 724. As an example, the IPD estimation module 724 may apply filtering, averaging over time, weighted averaging including assumed IPD distances, Kalman filtering, etc. as part of estimating the user's IPD in an accurate manner.

[0496] The vergence-divergence movement depth estimation module 728 may receive data from various modules and sub-modules (as shown in connection with FIG. 7A ) within the eye tracking module 614. In particular, the vergence-divergence movement depth estimation module 728 may employ data indicative of an estimated 3D position of the pupil center (e.g., as provided by module 720 described above), one or more determined parameters of the optical axis (e.g., as provided by module 722 described above), an estimated 3D position of the center of rotation (e.g., as provided by module 724 described above), an estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (e.g., as provided by module 726 described above), and / or one or more determined parameters of the optical axis and / or visual axis (e.g., as provided by module 722 and / or module 730 described below). The convergence depth estimation module 728 may detect or otherwise obtain a measurement of the user's convergence depth, which may be the distance from the user at which the user's eyes are focused. As an example, when the user is looking at an object three feet in front of them, the user's left and right eyes have a convergence depth of three feet, while when the user is looking at a distant scene (e.g., the optical axes of the user's eyes are approximately parallel to one another such that the distance between the centers of the user's pupils may be approximately equal to the distance between the centers of rotation of the user's left and right eyes), the user's left and right eyes have a convergence depth of infinity. In some implementations, the convergence depth estimation module 728 may utilize data indicative of estimated centers of the user's pupils (e.g., as provided by module 720) and determine the 3D distance between the estimated centers of the user's pupils. The convergence-divergence depth estimation module 728 may obtain a measure of convergence-divergence depth by comparing such determined 3D distance between pupil centers with an estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (e.g., as shown by module 726 described above).In addition to the 3D distance between pupil centers and the estimated IPD, the convergence movement depth estimation module 728 may utilize known, assumed, estimated, and / or determined geometric shapes to calculate the convergence movement depth. As an example, the module 728 may combine the 3D distance between pupil centers, the estimated IPD, and the 3D CoR position in a trigonometric calculation to estimate (e.g., determine) the user's convergence movement depth. Indeed, an evaluation of such determined 3D distance between pupil centers relative to the estimated IPD may serve as an indication of the user's current convergence movement depth relative to optical infinity. In some examples, the convergence movement depth estimation module 728 may simply receive or have access to data indicating the estimated 3D distance between the estimated centers of the user's pupils for purposes of obtaining such a measurement of the convergence movement depth. In some embodiments, the vergence movement depth estimation module 728 may estimate the vergence movement depth by comparing the user's left and right optical axes. In particular, the vergence movement depth estimation module 728 may estimate the vergence movement depth by locating the distance from the user where the user's left and right optical axes intersect (or where projections of the user's left and right optical axes on a plane, such as a horizontal plane, intersect). The module 728 may utilize the user's IPD in this calculation by setting zero depth to be the depth where the user's left and right optical axes are separated by the user's IPD. In at least some embodiments, the vergence movement depth estimation module 728 may determine the vergence movement depth by triangulating eye tracking data with known or derived spatial relationships.

[0497] In some embodiments, the convergence-divergence depth estimation module 728 may estimate the user's convergence depth based on the intersection of the user's visual axes (instead of their optical axes), which may provide a more accurate indication of the distance the user is focusing. In at least some embodiments, the eye tracking module 614 may include an optical axis / visual axis mapping module 730. As discussed in further detail in connection with FIG. 10 , a user's optical axis and visual axis are generally not aligned. The visual axis is the axis along which a person looks, while the optical axis is defined by the center of the person's lens and pupil and may run through the center of the person's retina. In particular, the user's visual axis is generally defined by the location of the user's fovea, which may be offset from the center of the user's retina, thereby resulting in different optical and visual axes. In at least some of these embodiments, the eye tracking module 614 may include an optical axis / visual axis mapping module 730. The optical axis / visual axis mapping module 730 may correct for differences between the user's optical axis and visual axis and provide information about the user's visual axis to other components in the wearable system, such as the vergence-divergence depth estimation module 728 and the light field rendering controller 618. In some examples, the module 730 may use assumed eye dimensions 704 that include a typical offset of approximately 5.2° inward (nasally, toward the user's nose) between the optical axis and the visual axis. In other words, the module 730 may shift the user's left optical axis 5.2° nasally (toward the user's nose) to the right and the user's right optical axis 5.2° nasally (toward the left) to estimate the direction of the user's left and right optical axes. In other examples, the module 730 may utilize per-user calibration data 706 when mapping the optical axis (e.g., as shown by module 722 described above) to the visual axis. As an additional example, module 730 may shift the user's optical axis nasally by 4.0° to 6.5°, 4.5° to 6.0°, 5.0° to 5.4°, etc., or any range formed by any of these values.In some arrangements, module 730 may apply the shift based, at least in part, on characteristics of the particular user, such as their age, gender, vision prescription, or other relevant characteristics, and / or may apply the shift based, at least in part, on a calibration process for the particular user (e.g., to determine the particular user's optical axis-visual axis offset). In at least some embodiments, module 730 may also shift the origins of the left and right optical axes to correspond to the user's CoP (as determined by module 732) instead of the user's CoR.

[0498] An optional center of perspective (CoP) estimation module 732, when provided, may estimate the location of the user's left and right centers of perspective (CoP). The CoP is a useful location for a wearable system and, in at least some embodiments, may be a location directly in front of the pupil. In at least some embodiments, the CoP estimation module 732 may estimate the location of the user's left and right centers of perspective based on the 3D location of the user's pupil center, the 3D location of the user's corneal curvature center, or any such suitable data, or any combination thereof. As an example, the user's CoP may be approximately 5.01 mm in front of the center of corneal curvature (e.g., 5.01 mm from the center of the corneal sphere, toward the cornea of ​​the eye, in a direction along the optical axis) and approximately 2.97 mm behind the outer surface of the user's cornea along the optical or visual axis. The user's center of perspective may be directly in front of the center of their pupil. As examples, the user's CoP may be less than about 2.0 mm from the user's pupil, less than about 1.0 mm from the user's pupil, or less than about 0.5 mm from the user's pupil, or any range between any of these values. As another example, the center of gaze may correspond to a location within the anterior chamber of the eye. As other examples, the CoP may be between 1.0 mm and 2.0 mm, about 1.0 mm, 0.25 mm and 1.0 mm, 0.5 mm and 1.0 mm, or 0.25 mm and 0.5 mm.

[0499] The viewpoint centers described herein (as potentially desirable locations for the rendering camera's pinhole and anatomical locations within the user's eye) may be locations that serve to reduce and / or eliminate undesirable parallax shift. In particular, the optical system of the user's eye roughly parallels a theoretical system formed by a pinhole in front of a lens projecting onto a screen, with the pinhole, lens, and screen roughly corresponding to the user's pupil / iris, lens, and retina, respectively. Furthermore, it may be desirable for there to be little or no parallax shift when two point light sources (or objects) at different distances from the user's eye are rotated precisely around the pinhole opening (e.g., rotated along radii of curvature equal to their respective distances from the pinhole opening). Thus, one would think that the CoP should be located at the center of the eye's pupil (and such a CoP may be used in some embodiments). However, in addition to the lens and pupil pinhole, the human eye includes a cornea, which imparts additional refractive power to light propagating toward the retina. Thus, the anatomical equivalent of a pinhole in the theoretical system described in this paragraph may be a region of a user's eye located between the outer surface of the cornea of ​​the user's eye and the center of the pupil or iris of the user's eye. For example, the anatomical equivalent of a pinhole may correspond to a region within the anterior chamber of the user's eye. For various reasons discussed herein, it may be desirable to set the CoP at such a location within the anterior chamber of the user's eye. The derivation and significance of the CoP are described in more detail below with respect to Figures 22-24B.

[0500] As discussed above, the eye tracking module 614 may provide data such as estimated 3D positions of the left and right eye centers of rotation (CoR), vergence and divergence movement depth, left and right eye optical axes, 3D positions of the user's eyes, 3D positions of the user's left and right centers of corneal curvature, 3D positions of the user's left and right pupil centers, 3D positions of the user's left and right gaze centers, and the user's IPD to other components in the wearable system, such as the light field rendering controller 618 and the alignment observer 620. The eye tracking module 614 may also include other sub-modules that detect and generate data associated with other aspects of the user's eyes. As an example, the eye tracking module 614 may include an eye blink detection module that provides a flag or other alert whenever the user blinks, and a saccade detection module that provides a flag or other alert whenever the user's eyes saccade (e.g., rapidly shift focus to another point).

[0501] Rendering Controller Example A detailed block diagram of an exemplary light field rendering controller 618 is shown in FIG. 7B. As shown in FIGS. 6 and 7B, the rendering controller 618 may receive eye tracking information from the eye tracking module 614 and provide output to the rendering engine 622, which may generate images to be displayed for viewing by a user of the wearable system. As an example, the rendering controller 618 may receive other eye data such as vergence and divergence depth, left and right eye rotation centers (and / or gaze centers), and eye blink data, saccade data, etc.

[0502] The depth plane selection module 750 may receive convergence-divergence depth information and other ocular data and, based on such data, may cause the rendering engine 622 to convey content to the user at a particular depth plane (e.g., a particular accommodation or focal length). As discussed in connection with FIG. 4 , the wearable system may include multiple discrete depth planes formed by multiple waveguides, each conveying image information with a variable level of wavefront curvature. In some embodiments, the wearable system may include one or more variable depth planes, such as optical elements, conveying image information with a time-varying level of wavefront curvature. In these and other embodiments, the depth plane selection module 750 may cause the rendering engine 622 to convey content to the user at a selected depth (e.g., cause the rendering engine 622 to instruct the display 220 to switch depth planes) based, in part, on the user's convergence-divergence depth. In at least some embodiments, depth plane selection module 750 and rendering engine 622 may render content at different depths and may also generate and / or provide depth plane selection data to display hardware, such as display 220. Display hardware, such as display 220, may perform electronic depth plane switching in response to depth plane selection data (which may be control signals) generated and / or provided by modules, such as depth plane selection module 750 and rendering engine 622.

[0503] In general, it may be desirable for depth plane selection module 750 to select a depth plane that matches the user's current convergence-divergence depth so that the user is provided with accurate accommodation cues. However, it may also be desirable to switch depth planes in a discreet and unobtrusive manner. As an example, it may be desirable to avoid excessive switching between depth planes and / or to switch depth planes at times when the user is unlikely to notice the switch, such as during an eyeblink or eye saccade.

[0504] The hysteresis band crossing detection module 752 may help avoid excessive switching between depth planes, particularly when the user's convergence-divergence depth fluctuates at the midpoint or transition point between two depth planes. In particular, module 752 may cause the depth plane selection module 750 to exhibit hysteresis in its selection of depth planes. As an example, module 752 may cause the depth plane selection module 750 to switch from a first, more distant depth plane to a second, closer depth plane only after the user's convergence-divergence depth passes a first threshold. Similarly, module 752 may cause the depth plane selection module 750 (and thus may indicate on a display, such as display 220) to switch to the first, more distant depth plane only after the user's convergence-divergence depth passes a second threshold that is farther from the user than the first threshold. In the overlap region between the first and second thresholds, module 750 may cause depth plane selection module 750 to maintain either depth plane as currently selected as the selected depth plane, thus avoiding excessive switching between depth planes.

[0505] The eye event detection module 750 may receive other eye data from the eye tracking module 614 of FIG. 7A and may cause the depth plane selection module 750 to delay some depth plane switches until an eye event occurs. As an example, the eye event detection module 750 may cause the depth plane selection module 750 to delay a planned depth plane switch until a user blink is detected, or may receive data from an eye blink detection component within the eye tracking module 614 indicating that the user is currently blinking, and in response, cause the depth plane selection module 750 to perform a planned depth plane switch during the blink event (e.g., by having the module 750 instruct the display 220 to perform a depth plane switch during the blink event). In at least some embodiments, the wearable system may be able to shift content onto a new depth plane during the blink event such that the user is unlikely to perceive the shift. As another example, the eye event detection module 750 may delay a planned depth plane switch until an eye saccade is detected. As discussed in relation to eye blinking, such an arrangement can facilitate discrete shifts in the depth plane.

[0506] If desired, the depth plane selection module 750 may delay a planned depth plane switch for only a limited period of time before executing a depth plane switch, even in the absence of an eye event. Similarly, the depth plane selection module 750 may execute a depth plane switch when the user's convergence-divergence depth is substantially outside the currently selected depth plane (e.g., when the user's convergence-divergence depth exceeds a predetermined threshold that exceeds the normal threshold for a depth plane switch), even in the absence of an eye event. These arrangements may help ensure that the eye event detection module 754 does not delay a depth plane switch indefinitely and does not delay a depth plane switch when a large accommodation error is present. Further details of the operation of the depth plane selection module 750 and how the module may time a depth plane switch are provided herein in connection with FIG. 12.

[0507] The rendering camera controller 758 may provide information indicating the locations of the user's left and right eyes to the rendering engine 622. The rendering engine 622 may then generate content by simulating cameras at the locations of the user's left and right eyes and generating content based on the perspectives of the simulated cameras. As discussed above, the rendering camera is a simulated camera for use in rendering virtual image content, possibly from a database of objects in the virtual world. The objects may have a location and orientation relative to the user or wearer, and possibly relative to real objects in the environment surrounding the user or wearer. The rendering camera may render virtual images based on a database of virtual objects contained within the rendering engine to be presented to the eyes. The virtual images may be rendered as if they were captured from the perspective of the user or wearer. For example, the virtual images may be rendered as if they were captured by a camera (corresponding to a "rendering camera") having an aperture, lens, and detector that views objects in the virtual world. The virtual images are captured from the perspective of such a camera, which has the location of the "rendering camera." For example, the virtual image may be rendered as if captured from the viewpoint of a camera having a specific location relative to the user's or wearer's eyes, to provide an image that appears to be from the user's or wearer's point of view. In some implementations, the image is rendered as if captured from the viewpoint of a camera having an aperture at a specific location relative to the user's or wearer's eyes (such as a viewpoint center or rotation center or other location as discussed herein).

[0508] The rendering camera controller 758 may determine the positions of the left and right cameras based on the left and right eye centers of rotation (CoR) determined by the CoR estimation module 724 and / or based on the left and right eye centers of viewpoint (CoP) determined by the CoP estimation module 732. In some embodiments, the rendering camera controller 758 may switch between the CoR and CoP locations based on various factors. As examples, the rendering camera controller 758 may, in various modes, always align the rendering camera to the CoR location, always align the rendering camera to the CoP location, toggle or discretely switch between aligning the rendering camera to the CoR location and aligning the rendering camera to the CoP location over time based on various factors, or dynamically align the rendering camera to any of a range of different positions along the optical axis (or visual axis) between the CoR and CoP locations over time based on various factors. The CoR and CoP positions may optionally be passed through a smoothing filter 756 (in any of the aforementioned modes for rendering camera positioning), which may average the CoR and CoP locations over time to reduce noise in these positions and prevent jitter when rendering the simulated rendering camera.

[0509] In at least some embodiments, the rendering camera may be simulated as a pinhole camera with the pinhole placed at the location of the estimated CoR or CoP identified by the eye tracking module 614. Because the CoP is offset from the CoR, whenever the rendering camera's position is based on the user's CoP, the locations of both the rendering camera and its pinhole will shift as the user's eyes rotate (see, e.g., how the rendering camera translates linearly with eye rotation as shown in FIGS. 16A and 16B). In contrast, whenever the rendering camera's position is based on the user's CoR, the location of the rendering camera's pinhole does not move with eye rotation, although the rendering camera (behind the pinhole) may move with eye rotation in some embodiments. In other embodiments where the rendering camera's position is based on the user's CoR, the rendering camera may not move (i.e., rotate) with the user's eyes (see, e.g., how the rendering camera does not move or translate linearly with eye rotation as depicted in FIGS. 17A and 17B).

[0510] Example of Locating a User's Cornea Using an Eye Tracking System 8A is a schematic diagram of an eye showing the spherical cornea of ​​the eye. As shown in FIG. 8A, a user's eye 810 may have a cornea 812, a pupil 822, and a lens 820. The cornea 812 may have a generally spherical shape, as indicated by a spherical corneal surface 814. The spherical corneal surface 814 may have a center point 816, also referred to as the corneal center, and a radius 818. The hemispherical cornea of ​​the user's eye may curve around the corneal center 816.

[0511] 8B-8E illustrate an example of using the 3D corneal center estimation module 716 and the eye tracking module 614 to locate the corneal center 816 of a user.

[0512] 8B , the 3D corneal center estimation module 716 may receive an eye tracking image 852, including a corneal flash 854. The 3D corneal center estimation module 716 may then simulate the known 3D positions of the eye camera 324 and light source 326 (which may be based on data in the eye tracking extrinsic and intrinsic properties database 702, the assumed eye dimensions database 704, and / or the per-user calibration data 706) in the eye camera coordinate system 850 to project a ray 856 into the eye camera coordinate system. In at least some embodiments, the eye camera coordinate system 850 may have its origin at the 3D position of the eye tracking camera 324.

[0513] 8C, the 3D corneal center estimation module 716 simulates a corneal sphere 814a (which may be based on assumed eye dimensions from the database 704) and a corneal center of curvature 816a at a first position. The 3D corneal center estimation module 716 may then check whether the corneal sphere 814a will properly reflect light from the light source 326 to the glint position 854. As shown in FIG. 8C, the first position does not match because the light ray 860a does not intersect with the light source 326.

[0514] Similar to Figure 8D, the 3D corneal center estimation module 716 simulates the corneal sphere 814b and corneal center of curvature 816b at a second position. The 3D corneal center estimation module 716 then checks whether the corneal sphere 814b properly reflects light from the light source 326 to the glint position 854. As shown in Figure 8D, the second position also does not match.

[0515] 8E, the 3D corneal center estimation module 716 can ultimately determine that the correct location of the corneal sphere is corneal sphere 814c and corneal center of curvature 816c. The 3D corneal center estimation module 716 verifies that the illustrated location is correct by checking that light from source 326 will properly reflect off the corneal sphere and be imaged by camera 324 at the correct location of flash 854 on image 852. Using this arrangement and the known 3D positions of light source 326, camera 324, and the camera's optical properties (such as focal length), the 3D corneal center estimation module 716 can determine the 3D location (relative to the wearable system) of the corneal center of curvature 816.

[0516] The process described herein with respect to at least FIGS. 8C-8E may effectively be an iterative, iterative, or optimization process for identifying the 3D location of the user's corneal center. Accordingly, any of a number of techniques (e.g., iterative, optimization, etc.) may be used to efficiently and quickly filter or reduce the search space of possible locations. Furthermore, in some embodiments, the system may include two, three, four, or more light sources, such as light source 326, some of which may be positioned at different locations and result in multiple photes, such as photes 854, located at different locations on image 852, and multiple light rays, such as ray 856, having different origins and directions. Such an embodiment may improve the accuracy of 3D corneal center estimation module 716, because module 716 may seek to identify a corneal position that results in some or all of the photes and light rays being properly reflected between that individual light source and that individual location on image 852. In other words, in these embodiments, the position of some or all of the light sources may rely on the 3D corneal position determination (e.g., iterative, optimization techniques, etc.) process of Figures 8B-8E. In some implementations, the system may determine a vector or ray (i.e., a 2D corneal center position) along which the center of the cornea resides before performing the optimization process. In such implementations, the 3D corneal center estimation module 716 may search only for corneal positions along such vectors, which may serve to provide computational and / or time savings when performing the optimization process. In at least some of these implementations, before determining such a vector, the system may first (i) define a first plane between the origin of the eye camera coordinate system 850, the first light source (e.g., light source 326a), and the first flash of light produced by the first light source (e.g., flash 854a), and (ii) define a second plane between the origin of the eye camera coordinate system 850, the second light source (e.g., light source 326b), and the second flash of light produced by the second light source (e.g., flash 854b).The system may then simply calculate the cross product of the first and second planes and determine the vector or ray along which the center of the cornea resides (ie, the 2D corneal center location).

[0517] Example of normalizing the coordinate system of eye-tracking images 9A-9C illustrate example normalization of the coordinate system of the eye tracking images by a component in the wearable system, such as coordinate system normalization module 718 of FIG. 7A. Normalization of the coordinate system of the eye tracking images relative to the user's pupil location can compensate for slippage of the wearable system relative to the user's face (e.g., headset slippage), and such normalization can establish a consistent orientation and distance between the eye tracking images and the user's eyes.

[0518] 9A, coordinate system normalization module 718 may receive estimated 3D coordinates 900 of the center of rotation of the user's cornea and may receive a non-normalized eye tracking image, such as image 852. Eye tracking image 852 and coordinates 900 may be in a non-normalized coordinate system 850, based, as an example, on the location of eye tracking camera 324.

[0519] As a first normalization step, coordinate system normalization module 718 may rotate coordinate system 850 into rotated coordinate system 902, as shown in Figure 9B, so that the z-axis (e.g., vergence-divergence depth axis) of the coordinate system may be aligned with the vector between the origin of the coordinate system and corneal center of curvature coordinate 900. In particular, coordinate system normalization module 718 may rotate eye tracking image 850 into rotated eye tracking image 904 until the user's corneal center of curvature coordinate 900 is normal to the plane of rotated image 904.

[0520] As a second normalization step, coordinate system normalization module 718 may translate rotated coordinate system 902 into normalized coordinate system 910 so that corneal curvature center coordinate 900 is at a standard normalized distance 906 from the origin of normalized coordinate system 910, as shown in FIG. 9C . In particular, coordinate system normalization module 718 may translate rotated eye tracking image 904 into normalized eye tracking image 912. In at least some embodiments, standard normalized distance 906 may be approximately 30 millimeters. If desired, the second normalization step may be performed prior to the first normalization step.

[0521] Example of locating a user's pupil centroid using an eye tracking system 9D-9G illustrate examples of using the 3D pupil center locator module 720 and the eye tracking module 614 to locate the user's pupil center (e.g., the center of the user's pupil 822 as shown in FIG. 8A).

[0522] 9D , 3D pupil center locator module 720 may receive a normalized eye tracking image 912 that includes a pupil centroid 913 (e.g., the center of the user's pupil as identified by pupil identification module 712). 3D pupil center locator module 720 may then simulate a normalized 3D position 910 of the eye camera 324 and project a ray 914 through the pupil centroid 913 into the normalized coordinate system 910.

[0523] 9E, 3D pupil center locator module 720 may simulate a corneal spherical surface, such as corneal spherical surface 901 having a center of curvature 900, based on data from 3D corneal center estimation module 716 (and as discussed in more detail in connection with FIGS. 8B-8E). As an example, corneal spherical surface 901 may be positioned within normalized coordinate system 910 based on the location of center of curvature 816c identified in connection with FIG. 8E and based on the normalization process of FIGS. 9A-9C. In addition, 3D pupil center locator module 720 may identify a first intersection point 916 between ray 914 (e.g., a ray between the origin of normalized coordinate system 910 and the normalized location of the user's pupil) and the simulated cornea, as shown in FIG. 9E.

[0524] As shown in FIG. 9F , the 3D pupil center locator module 720 may determine a pupil sphere 918 based on the corneal sphere 901. The pupil sphere 918 may share a common center of curvature with the corneal sphere 901 but have a smaller radius. The 3D pupil center locator module 720 may determine the distance between the corneal center 900 and the pupil sphere 918 (e.g., the radius of the pupil sphere 918) based on the distance between the corneal center and the pupil center. In some embodiments, the distance between the pupil center and the corneal center of curvature may be determined from the assumed eye dimensions 704 of FIG. 7A , from the eye tracking extrinsic and intrinsic properties database 702, and / or from the per-user calibration data 706. In other embodiments, the distance between the pupil center and the corneal center of curvature may be determined from the per-user calibration data 706 of FIG. 7A .

[0525] 9G , 3D pupil center locator module 720 may locate the 3D coordinates of the user's pupil center based on various inputs. As an example, 3D pupil center locator module 720 may determine the 3D coordinates of the user's pupil center using the 3D coordinates and radius of pupil sphere 918, the 3D coordinates of intersection 916 between simulated corneal sphere 901 and ray 914 associated with pupil centroid 913 in normalized eye tracking image 912, information about the refractive index of the cornea, and other relevant information, such as the refractive index of air (which may be stored in eye tracking extrinsic and intrinsic properties database 702). In particular, 3D pupil center locator module 720 may bend ray 916 into refracted ray 922 in the simulation based on the refractive index difference between air (at a first refractive index of approximately 1.00) and corneal material (at a second refractive index of approximately 1.38). After accounting for refraction caused by the cornea, the 3D pupil center locator module 720 may determine the 3D coordinate of a first intersection point 920 between the refracted light ray 922 and the pupil sphere 918. The 3D pupil center locator module 720 may determine that the user's pupil center 920 is located approximately at the first intersection point 920 between the refracted light ray 922 and the pupil sphere 918. Using this arrangement, the 3D pupil center locator module 720 can determine the 3D location (relative to the wearable system) of the pupil center 920 within the normalized coordinate system 910. If desired, the wearable system can de-normalize the coordinates of the pupil center 920 to the original eye camera coordinate system 850. The pupil center 920 may be used in conjunction with the corneal curvature center 900 to determine, among other things, the user's optical axis using the optical axis determination module 722 and the user's convergence and divergence movement depth using the convergence and divergence movement depth estimation module 728.

[0526] Considering corneal refraction can potentially result in a more stable determined pupil position than one based on the first intersection point 916 between ray 914 (i.e., the ray between the origin of normalized coordinate system 910 and the normalized location of the user's pupil) and the simulated cornea, as shown in FIG. 9E. This is true in part because, although it is simpler to calculate, the first intersection point 916 may not correspond to a physical feature of the eye and therefore may not move with the eye as a fixed body. In contrast, a calculation of pupil center 920 that takes corneal refraction into account may better correspond to the eye's physical pupil position, even if there is still a certain amount of variation as a result of viewing angle. In various implementations, determining the eye's optical axis may therefore involve calculating the true pupil center rather than the first intersection point 916 between ray 914 and the simulated cornea.

[0527] A significant advantage of including corneal refraction occurs when the center of rotation (CoR) is estimated as a point at a fixed distance from the corneal center along the eye's optical axis. In particular, including corneal refraction in determining pupil position can significantly reduce variability in calculating the center of rotation for different eye orientations. For example, variability can occur when the eye moves as a whole within the camera coordinate frame, such as during headset remounting, because the eye may be oriented differently relative to the headset upon remounting. Because the pupil center 920 corresponds more closely to the eye's physical pupil position, there can be little variability in the CoR as the eye moves as a whole within the camera coordinate frame. Advantageously, including corneal surface refraction can result in a more stable and accurate CoR, potentially useful in determining when a headset is replaced on a user's head, enable more accurate rendering camera placement, enable other novel eye-tracking algorithms, or any combination thereof. Additionally, the CoR, as a stable, slowly changing feature of the eye, can potentially be tracked by a multi-frame Kalman-type temporal filter to provide a geometric reference location for other applications. FIG. 9H illustrates exemplary calculated locations of the pupil center from including and excluding corneal refraction. When the pupil center is calculated without including the effects of corneal refraction, an external pupil 960 may result. The external pupil center 960 may correspond to the first intersection 916 between ray 914 (i.e., the ray between the origin of the normalized coordinate system 910 and the normalized location of the user's pupil) and the simulated cornea, as shown in FIG. 9E. When the pupil center is calculated taking corneal refraction into account, a refracted pupil center 962 may result. The refracted pupil center 962 may correspond to the pupil center 920, as shown in FIG. 9G. Different locations of the external pupil center 960 and the refracted pupil center 962 may result in different calculations of the eye's center of rotation, as determined as a fixed distance along the eye's optical axis from the corneal center 964. For example, the external pupil center 960 may result in an external center of rotation 968. The external center of rotation 968 may vary significantly from the center of rotation 966 calculated from the refracted pupil center 962.

[0528] Figures 9I-9L illustrate exemplary experimental variability in the calculated center of rotation of the eye based on different pupil centers calculated using a collection of over 400 data sets. The data selected included only frames with four phosphenes and an effective pupil center (i.e., all three x, y, and z coordinate components not equal to zero) to eliminate obvious pipeline failures. Different center of rotation (CoR) to corneal curvature center distance values ​​(R) were explored, and R = 4.7 mm was found to provide near-optimal average results. However, specific user distance values ​​can be adjusted to provide even smaller variability in the CoR coordinate.

[0529] FIG. 9I illustrates a three-dimensional graph of x, y, and z coordinates of different pupil centers and the corresponding CoR calculated using the different pupil centers for the dataset described above. FIG. 9J illustrates the data of FIG. 9I in an XY projection. Cluster 970 corresponds to coordinates for external pupil center 960, cluster 972 corresponds to coordinates for correct refracted pupil center 962, cluster 974 corresponds to corneal locations (e.g., three-dimensional corneal center locations), cluster 978 corresponds to the CoR using correct refracted pupil center 962, and cluster 980 corresponds to the CoR using external pupil center 960. Cluster 978 is smaller in size, particularly in the x-direction, than cluster 980, indicating less variability in the CoR using refracted pupil center 962 than the CoR using external pupil center 960. [Table 1]

[0530] As can be seen from Table 1, the standard deviation (or Σ) of the x-component of CoR was reduced by approximately half when the refracted pupil center 962 was used for the calculation compared to when the external pupil center 960 was used. The total three-dimensional standard deviation (Σ 3d ) was also significantly reduced with the use of the refracted pupil center 962.

[0531] 9K and 9L respectively illustrate the mean and median CoR standard deviation as a function of distance from the CoR to the center of corneal curvature for a collection of over 400 data sets. Graphs 991L and 991R show the mean 3D CoR standard deviation (Σ 3d ) are shown. Graphs 992L and 992R show the median 3D CoR standard deviation (Σ 3d ) is shown. Curve 982A shows the total left eye average value Σ 3d Curve 982B corresponds to the total right eye average value Σ 3d Curve 982C corresponds to the total left eye median Σ 3d Curve 982D corresponds to the total right eye median Σ 3d Curves 984A-D correspond to various Σ 3d Curves 990A-D correspond to the x-component of various Σ 3d Curves 986A-D correspond to the y-component of various Σ 3d corresponds to the z-component of

[0532] Figure 9M shows the Σ 3d 1 illustrates an exemplary distribution of optimal radii calculated individually for each user data set as the value of the distance from CoR to the cornea that provides the smallest value of . The mean value of the distribution was found to be R = 4.9 mm, with a standard deviation of 1.5 mm. Users with very small radii (R ≈ 1 mm) were found to be examples with very poor eye tracking and therefore had to be excluded.

[0533] Example of the difference between the optical axis and the visual axis As discussed in connection with the optical axis / visual axis mapping module 730 of FIG. 7A , a user's optical axis and visual axis are generally not aligned, in part because the user's visual axis is defined by their fovea, which is generally not at the center of the person's retina. Thus, when a person desires to focus on a particular object, the person aligns their visual axis with the object, ensuring that light from the object falls on their fovea, while their optical axis (defined by the center of their pupil and the center of curvature of their cornea) is actually slightly offset from the object. FIG. 10 is an example of an eye 1000 illustrating the eye's optical axis 1002, the eye's visual axis 1004, and the offset between these axes. Additionally, FIG. 10 illustrates the eye's pupil center 1006, the eye's corneal center of curvature 1008, and the eye's mean center of rotation (CoR) 1010. In at least some populations, the eye's center of corneal curvature 1008 may be approximately 4.7 mm in front of the eye's mean center of rotation (CoR) 1010, as indicated by dimension 1012. Additionally, the eye's center of gaze 1014 may be approximately 5.01 mm in front of the eye's center of corneal curvature 1008, approximately 2.97 mm behind the outer surface 1016 of the user's cornea, and / or directly in front of the user's pupil center 1006 (e.g., corresponding to a location within the anterior chamber of the eye 1000). As additional examples, dimension 1012 may be between 3.0 mm and 7.0 mm, between 4.0 and 6.0 mm, between 4.5 and 5.0 mm, or between 4.6 and 4.8 mm, or any range between any value within any of these ranges. The eye's center of perspective (CoP) 1014 can be a useful location for wearable systems because, in at least some embodiments, aligning the rendering camera to the CoP can help reduce or eliminate parallax artifacts.

[0534] 10 also illustrates such locations within the human eye 1000 with which the rendering camera pinhole may be aligned. As shown in FIG. 10 , the rendering camera pinhole may be aligned with a location 1014 along the optical axis 1002 or visual axis 1004 of the human eye 1000 that is closer to the outer surface of the cornea than both (a) the center of the pupil or iris 1006 and (b) the center of corneal curvature 1008 of the human eye 1000. For example, as shown in FIG. 10 , the rendering camera pinhole may be aligned with a location 1014 along the optical axis 1002 of the human eye 1000 that is approximately 2.97 millimeters posterior to the outer surface of the cornea 1016 and approximately 5.01 millimeters anterior to the center of corneal curvature 1008. The location 1014 of the rendering camera's pinhole and / or the anatomical region of the human eye 1000 to which the location 1014 corresponds may be considered to represent the center of view of the human eye 1000. The optical axis 1002 of the human eye 1000 as shown in Figure 10 represents the shortest line passing through the center of corneal curvature 1008 and the center of the pupil or iris 1006. The visual axis 1004 of the human eye 1000 differs from the optical axis 1002 because it represents a line extending from the fovea of ​​the human eye 1000 to the center of the pupil or iris 1006. Exemplary Process for Rendering Content and Checking Alignment Based on Eye Tracking 11 is a process flow diagram of an example method 1100 for using eye tracking to provide feedback regarding alignment within a wearable device when rendering content. Method 1100 may be implemented by a wearable system described herein. An embodiment of method 1100 can be used by a wearable system to render content and provide feedback regarding alignment (e.g., the fit of the wearable device with a user) based on data from the eye tracking system.

[0535] In block 1110, the wearable system may capture images of one or both of the user's eyes. The wearable system may capture the eye images using one or more eye cameras 324, as shown in at least the example of FIG. 3. Optionally, the wearable system may also include one or more light sources 326 configured to shine IR light on the user's eyes and produce corresponding flashes of light in the eye images captured by the eye cameras 324. As discussed herein, the flashes of light may be used by the eye tracking module 614 to derive various information about the user's eyes, including where the eyes are looking.

[0536] In block 1120, the wearable system may detect flashes of light and pupils in the eye image captured in block 1110. As an example, block 1120 may include processing the eye image by flash detection and labeling module 714 to identify a two-dimensional location of the flash of light in the eye image, and processing the eye image by pupil identification module 712 to identify a two-dimensional location of the pupil in the eye image.

[0537] In block 1130, the wearable system may estimate the three-dimensional positions of the user's left and right corneas relative to the wearable system. As an example, the wearable system may estimate the locations of the centers of curvature of the user's left and right corneas and the distances between those centers of curvature and the user's left and right corneas. Block 1130 may involve a 3D corneal center estimation module 716 that identifies the locations of the centers of curvature, as described herein in connection with at least Figures 7A and 8A-8E.

[0538] In block 1140, the wearable system may estimate the three-dimensional location of the user's left and right pupil centers relative to the wearable system. As an example, the wearable system and 3D pupil center locator module 720 may estimate the location of the user's left and right pupil centers as part of block 1140, particularly as described in connection with at least Figures 7A and 9D-9G.

[0539] In block 1150, the wearable system may estimate the three-dimensional location of the user's left and right center or rotation (CoR) relative to the wearable system. As an example, the wearable system and CoR estimation module 724 may estimate the location of the CoR for the user's left and right eyes, particularly as described in connection with at least FIGS. 7A and 10. As a particular example, the wearable system may find the CoR of the eye by tracing backward along the optical axis from the center of curvature of the cornea toward the retina.

[0540] In block 1160, the wearable system may estimate the user's IPD, convergence depth, center of gaze (CoP), optical axis, visual axis, and other desired attributes from the eye tracking data. As an example, as part of block 1160, the IPD estimation module 726 may estimate the user's IPD by comparing the 3D positions of the left and right CoR, the convergence depth estimation module 728 may estimate the user's depth by finding the intersection (or near the intersection) of the left and right optical axes or the intersection of the left and right visual axes, the optical axis determination module 722 may identify the left and right optical axes over time, the optical axis / visual axis mapping module 730 may identify the left and right visual axes over time, and the CoP estimation module 732 may identify the left and right centers of gaze.

[0541] In block 1170, the wearable system may render content and, optionally, provide feedback regarding alignment (e.g., the fit of the wearable system with the user's head) based in part on the eye-tracking data identified in blocks 1120-1160. As an example, the wearable system may identify a preferred location for a rendering camera, as discussed in connection with light field rendering controller 618 ( FIG. 7B ) and rendering engine 622, and then generate content for the user based on the location of the rendering camera. As another example, the wearable system may determine whether it is properly fitted to the user or has slipped out of its proper place relative to the user, as discussed in connection with alignment observer 620, and may provide optional feedback to the user indicating whether the fit of the device needs adjustment. In some embodiments, the wearable system may adjust rendered content based on improper or sub-ideal alignment in an attempt to reduce, minimize, or compensate for the effects of the improper or misaligned alignment.

[0542] Illustrative Graph for Rendering Content in Responsive to User Eye Movements 12 includes a set of example graphs 1200a-1200j illustrating how a wearable system may switch depth planes in response to a user's eye movement. As discussed herein in connection with FIGS. 4 and 7, the wearable system may include multiple depth planes, with the various depth planes configured to present content to the user at different simulated depths or with different accommodation cues (e.g., with various levels of wavefront curvature or ray divergence). As an example, the wearable system may include a first depth plane configured to simulate a first depth range and a second depth plane configured to simulate a second depth range, where these two ranges may desirably overlap to facilitate hysteresis in switching, although the second depth range may generally extend a greater distance from the user. In such embodiments, the wearable system may track the user's convergence depth, saccade movements, and eyeblinks, and switch between the first and second depth pl...

Claims

[Claim 1] The invention described in this specification.

Citation Information

Patent Citations

  • Sight line detecting system

    JP2005013752A

  • Head-mounted system and method for computing and rendering a stream of digital images using the head-mounted system

    JP2016502120A

  • Circular light element for illumination of cornea in head mounted eye-tracking

    US20160363995A1

  • Spectacle lens design system, supply system, design method, and production method

    WO2014133166A1